2024-03-29T10:41:03.521 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:41:03.522 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/orig.config.yaml 2024-03-29T10:41:03.558 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:41:03.645 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622660' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI4p6ibs7lJ1PiB5ycWMubtAd2eIXTbxBFYTi66d687eW0XvLop/s7OH1CsTcIyd817+s3Z3oSShrHyRMeRBVPs= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBVO7ruRl0xSWBFQuwtmzxUOeWSAOQUBI3q2WwjDZWo+Z/fgFJpv8+BTJQkRGHrN+748R3CHxJKpIbVn/j9Vn7g= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:41:03.645 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:41:03.646 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:41:03.646 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:41:03.648 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:41:03.672 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:41:03.673 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:41:03.673 INFO:teuthology.packaging:ref: None 2024-03-29T10:41:03.673 INFO:teuthology.packaging:tag: None 2024-03-29T10:41:03.673 INFO:teuthology.packaging:branch: main 2024-03-29T10:41:03.673 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:41:03.673 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:41:04.039 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:41:04.041 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:41:04.153 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:41:04.153 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:41:04.177 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:41:04.191 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:41:04.206 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:41:04.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:36:10.429673', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI4p6ibs7lJ1PiB5ycWMubtAd2eIXTbxBFYTi66d687eW0XvLop/s7OH1CsTcIyd817+s3Z3oSShrHyRMeRBVPs='} 2024-03-29T10:41:04.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:36:10.433228', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBVO7ruRl0xSWBFQuwtmzxUOeWSAOQUBI3q2WwjDZWo+Z/fgFJpv8+BTJQkRGHrN+748R3CHxJKpIbVn/j9Vn7g='} 2024-03-29T10:41:04.260 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:41:04.285 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:41:04.286 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:41:04.286 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:41:04.374 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3a9494c0d0>, signals=[15]) 2024-03-29T10:41:04.375 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:41:04.397 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:41:04.397 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T10:41:04.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:41:04.508 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T10:41:04.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:41:04.608 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:41:05.082 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-03-29T10:41:05.101 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-03-29T10:41:05.101 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-03-29T10:41:05.165 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-03-29T10:41:05.165 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-03-29T10:41:05.165 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-03-29T10:41:05.165 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:41:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:41:05.167 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-03-29T10:41:05.196 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-29T10:41:05.217 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-29T10:41:05.217 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-29T10:41:05.276 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-29T10:41:05.276 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-29T10:41:05.276 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-29T10:41:05.276 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:41:05.276 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-29T10:41:05.276 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:41:05.277 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-29T10:41:05.295 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:41:05.309 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:41:05.324 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:41:05.326 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:41:05.331 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:41:05.349 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:41:05.362 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:41:05.363 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:41:05.389 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:41:05.414 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:41:05.470 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:41:05.485 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:41:05.486 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:41:05.486 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:41:05.487 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:41:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:41:05.504 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-29T10:41:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:41:05.505 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-29T10:41:05.561 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:41:05.561 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-29T10:41:05.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-29T10:41:05.563 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:41:05.563 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-361.el9.x86_64 2024-03-29T10:41:05.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-03-29T10:41:07.648 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:41:07.752 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:41:09.048 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-29T10:41:09.193 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-29T10:41:10.067 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T10:41:13.880 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-29T10:41:15.573 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-29T10:41:17.217 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-03-29T10:41:19.698 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-29T10:41:21.116 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T10:41:21.510 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T10:41:22.878 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T10:41:22.944 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 16 kB 00:00 2024-03-29T10:41:26.681 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 21 MB 00:03 2024-03-29T10:41:27.850 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T10:41:30.115 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:41:30.115 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:41:30.210 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:41:30.222 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:41:30.222 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:41:30.222 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:41:30.222 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:41:30.222 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:41:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 76 M 2024-03-29T10:41:30.224 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-29T10:41:30.224 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:41:30.368 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:41:30.827 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 25 MB/s | 15 MB 00:00 2024-03-29T10:41:31.452 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T10:41:31.693 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 21 MB/s | 28 MB 00:01 2024-03-29T10:41:31.696 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:Total 52 MB/s | 76 MB 00:01 2024-03-29T10:41:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:41:31.945 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T10:41:32.050 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:41:32.050 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:41:32.879 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:41:32.879 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:41:34.327 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:41:34.327 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:41:34.414 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:41:34.425 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:41:34.425 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:41:34.425 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:41:34.425 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-03-29T10:41:34.426 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:41:34.427 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 76 M 2024-03-29T10:41:34.427 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 123 M 2024-03-29T10:41:34.427 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:41:34.630 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 946 kB/s | 192 kB 00:00 2024-03-29T10:41:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:41:35.446 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-29T10:41:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:35.938 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T10:41:36.472 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:41:36.475 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:36.475 INFO:teuthology.orchestra.run.smithi045.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-29T10:41:36.475 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:41:36.925 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:41:36.925 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:41:37.774 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:41:37.774 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:41:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:38.696 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:40.203 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:41:40.810 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:40.923 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:43.397 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:43.487 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:43.834 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:49.038 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:49.080 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:54.176 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:17.310 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:17.310 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:42:17.310 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:42:17.353 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:17.697 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:17.697 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:42:17.697 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:42:17.697 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:42:18.072 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:18.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:42:18.072 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:42:18.072 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:42:18.072 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:42:18.072 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:42:18.073 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:42:18.073 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:42:18.073 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:42:18.378 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:42:18.496 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:42:18.496 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:42:18.496 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:42:18.496 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:42:18.496 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:42:18.497 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:42:18.497 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:42:18.497 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:42:18.577 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:42:18.577 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:42:18.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-29T10:42:18.611 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-29T10:42:18.611 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:42:18.612 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:42:18.613 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:42:18.614 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:42:18.614 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:42:18.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:42:19.280 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-29T10:42:20.667 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:42:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-29T10:42:20.698 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-29T10:42:20.713 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.25ytDuDRVe 2024-03-29T10:42:20.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.25ytDuDRVe 2024-03-29T10:42:20.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.25ytDuDRVe 2024-03-29T10:42:20.924 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.25ytDuDRVe is 6KB 2024-03-29T10:42:20.944 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.25ytDuDRVe 2024-03-29T10:42:20.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:42:21.030 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:42:21.030 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:42:21.031 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:42:21.031 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:42:21.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:42:21.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-29T10:42:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:25.140 INFO:teuthology.orchestra.run.smithi045.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:42:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:42:25.185 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:25.557 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:25.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:42:25.557 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:42:25.558 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:42:25.966 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:42:26.281 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:42:26.431 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:42:26.432 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:42:26.432 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:42:26.432 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:42:26.432 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:42:26.432 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:42:26.432 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:42:26.432 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:42:26.515 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:42:26.516 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:42:26.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-03-29T10:42:26.562 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:42:26.563 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:42:26.564 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:42:26.565 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:42:26.565 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:42:26.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:42:27.159 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-03-29T10:42:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:42:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-03-29T10:42:28.675 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-29T10:42:28.691 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.CYgBt9oHKI 2024-03-29T10:42:28.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CYgBt9oHKI 2024-03-29T10:42:28.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.CYgBt9oHKI 2024-03-29T10:42:28.906 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.CYgBt9oHKI is 6KB 2024-03-29T10:42:28.954 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.CYgBt9oHKI 2024-03-29T10:42:28.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:42:29.039 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:42:29.039 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:42:29.039 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:42:29.039 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:42:29.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:42:29.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-03-29T10:42:51.218 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:42:51.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-29T10:42:51.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:59.187 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:42:59.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T10:42:59.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:09.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-29T10:43:17.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-03-29T10:43:18.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-29T10:43:18.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:26.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T10:43:26.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:18.783 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:44:26.678 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:44:30.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-29T10:44:30.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:31.264 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-29T10:44:31.622 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-29T10:44:31.622 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:44:31.623 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-29T10:44:31.639 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:44:31.639 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:44:31.639 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:44:31.639 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:44:32.640 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:44:32.641 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:44:32.718 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-29T10:44:32.752 DEBUG:teuthology.parallel:result is None 2024-03-29T10:44:38.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T10:44:38.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:39.048 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-29T10:44:39.419 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T10:44:39.420 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:44:39.420 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-29T10:44:39.435 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:44:39.436 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:44:39.436 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:44:39.436 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:44:40.437 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:44:40.437 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:44:40.515 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-03-29T10:44:40.548 DEBUG:teuthology.parallel:result is None 2024-03-29T10:44:40.549 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:44:40.556 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:44:40.557 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:44:40.559 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:44:40.577 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:44:40.584 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:44:40.591 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:44:40.591 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:44:40.616 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:44:40.649 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:44:40.656 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:44:40.656 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:44:40.678 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:44:40.715 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:40.721 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:40.729 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:40.734 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:40.736 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:44:40.743 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:44:40.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:44:40.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:44:40.807 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:44:40.816 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:44:40.816 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:44:40.841 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:44:40.864 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:40.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:40.994 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:41.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:41.121 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:44:41.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:44:41.190 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:41.227 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:41.285 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:41.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:41.414 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:44:41.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:44:41.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-03-29T10:44:41.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-29T10:44:41.533 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:44:41.569 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:44:41.990 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:44:41.998 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:44:41.998 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:44:42.008 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:44:42.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-03-29T10:44:42.064 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-03-29T10:44:42.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-29T10:44:42.118 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-29T10:44:42.120 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:44:42.120 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-03-29T10:44:42.143 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-03-29T10:44:42.143 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-29T10:44:42.178 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-29T10:44:42.178 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:44:42.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:44:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:44:42.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:44:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:44:42.247 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:44:42.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:44:42.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:44:42.321 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:44:42.332 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:44:42.337 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:44:42.463 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:44:42.484 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:44:42.485 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:48:58.438 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2024-03-29T10:48:58.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T10:48:58.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:48:58.536 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-29T10:48:58.614 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T10:48:58.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-29T10:48:58.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:48:58.704 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-29T10:48:58.781 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-29T10:48:58.781 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:48:58.791 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:48:58.791 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:48:58.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:48:58.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:48:58.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:48:58.823 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:48:58.840 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:48:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:48:58.869 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:48:58.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-03-29T10:48:58.886 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-03-29T10:48:58.898 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-29T10:48:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:48:58.912 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-29T10:48:58.916 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:48:58.928 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:48:58.944 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:48:58.966 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:48:58.970 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:48:58.970 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-29T10:48:58.970 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:58.970 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:58.970 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:58.971 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:48:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:48:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-29T10:48:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:59.011 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:48:59.020 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:48:59.020 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:48:59.020 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:48:59.020 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:48:59.026 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:48:59.026 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:48:59.026 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:48:59.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:48:59.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:48:59.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:48:59.176 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:48:59.176 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:48:59.180 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:48:59.180 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:48:59.437 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:48:59.438 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:48:59.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:48:59.440 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:48:59.440 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:48:59.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:48:59.468 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:48:59.468 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:48:59.469 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:48:59.469 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:48:59.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:48:59.537 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:48:59.616 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:48:59.617 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:48:59.682 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-03-29T10:48:59.683 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-29T10:48:59.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-03-29T10:48:59.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-29T10:48:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-29T10:48:59.944 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-03-29T10:48:59.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2024-03-29T10:48:59.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-03-29T10:49:01.015 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 109 kB/s | 82 kB 00:00 2024-03-29T10:49:01.133 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 96 kB/s | 82 kB 00:00 2024-03-29T10:49:01.557 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-29T10:49:01.675 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-29T10:49:02.049 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-29T10:49:02.234 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-29T10:49:02.535 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:49:02.694 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:49:03.910 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T10:49:04.136 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T10:49:06.151 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-03-29T10:49:06.518 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-29T10:49:08.986 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T10:49:10.935 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-03-29T10:49:13.795 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.1 MB 00:01 2024-03-29T10:49:15.247 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-29T10:49:15.736 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-03-29T10:49:17.159 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T10:49:17.320 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-29T10:49:19.971 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-03-29T10:49:22.299 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-29T10:49:24.980 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-29T10:49:25.000 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:49:25.012 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:49:25.012 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:49:25.012 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:49:25.012 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:49:25.012 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:49:25.012 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:49:25.013 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:49:25.014 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:49:25.015 INFO:teuthology.orchestra.run.smithi045.stdout:Install 30 Packages 2024-03-29T10:49:25.016 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-03-29T10:49:25.016 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:25.016 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 58 M 2024-03-29T10:49:25.016 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:49:26.899 INFO:teuthology.orchestra.run.smithi045.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 5.3 MB 00:01 2024-03-29T10:49:27.008 INFO:teuthology.orchestra.run.smithi045.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 25 kB 00:00 2024-03-29T10:49:27.316 INFO:teuthology.orchestra.run.smithi045.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.4 MB/s | 742 kB 00:00 2024-03-29T10:49:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:49:27.548 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:49:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:49:27.548 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:49:27.548 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:49:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:49:27.550 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:49:27.551 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout:Install 30 Packages 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2024-03-29T10:49:27.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:27.553 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 58 M 2024-03-29T10:49:27.553 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:49:27.608 INFO:teuthology.orchestra.run.smithi045.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.7 MB/s | 498 kB 00:00 2024-03-29T10:49:28.133 INFO:teuthology.orchestra.run.smithi045.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 6.5 MB/s | 20 MB 00:03 2024-03-29T10:49:28.209 INFO:teuthology.orchestra.run.smithi045.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 603 kB/s | 45 kB 00:00 2024-03-29T10:49:28.284 INFO:teuthology.orchestra.run.smithi045.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T10:49:28.367 INFO:teuthology.orchestra.run.smithi045.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.9 MB/s | 162 kB 00:00 2024-03-29T10:49:28.459 INFO:teuthology.orchestra.run.smithi045.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 320 kB 00:00 2024-03-29T10:49:28.543 INFO:teuthology.orchestra.run.smithi045.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T10:49:28.644 INFO:teuthology.orchestra.run.smithi045.stdout:(11/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.7 MB/s | 9.7 MB 00:03 2024-03-29T10:49:28.661 INFO:teuthology.orchestra.run.smithi045.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 851 kB/s | 99 kB 00:00 2024-03-29T10:49:28.786 INFO:teuthology.orchestra.run.smithi045.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-03-29T10:49:28.802 INFO:teuthology.orchestra.run.smithi045.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-29T10:49:28.828 INFO:teuthology.orchestra.run.smithi045.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T10:49:28.878 INFO:teuthology.orchestra.run.smithi045.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T10:49:28.904 INFO:teuthology.orchestra.run.smithi045.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T10:49:28.920 INFO:teuthology.orchestra.run.smithi045.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 997 kB/s | 41 kB 00:00 2024-03-29T10:49:28.946 INFO:teuthology.orchestra.run.smithi045.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-03-29T10:49:28.963 INFO:teuthology.orchestra.run.smithi045.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 838 kB/s | 35 kB 00:00 2024-03-29T10:49:28.980 INFO:teuthology.orchestra.run.smithi045.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T10:49:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 5.1 MB 00:01 2024-03-29T10:49:29.464 INFO:teuthology.orchestra.run.smithi045.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-03-29T10:49:29.489 INFO:teuthology.orchestra.run.smithi045.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 587 kB/s | 309 kB 00:00 2024-03-29T10:49:29.664 INFO:teuthology.orchestra.run.smithi116.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 5.3 MB 00:01 2024-03-29T10:49:29.697 INFO:teuthology.orchestra.run.smithi116.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 762 kB/s | 25 kB 00:00 2024-03-29T10:49:29.806 INFO:teuthology.orchestra.run.smithi045.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-29T10:49:29.831 INFO:teuthology.orchestra.run.smithi045.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-29T10:49:29.864 INFO:teuthology.orchestra.run.smithi116.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.3 MB/s | 742 kB 00:00 2024-03-29T10:49:29.998 INFO:teuthology.orchestra.run.smithi116.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.7 MB/s | 498 kB 00:00 2024-03-29T10:49:30.081 INFO:teuthology.orchestra.run.smithi045.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-03-29T10:49:30.340 INFO:teuthology.orchestra.run.smithi045.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-03-29T10:49:30.473 INFO:teuthology.orchestra.run.smithi045.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-29T10:49:30.840 INFO:teuthology.orchestra.run.smithi045.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-29T10:49:30.948 INFO:teuthology.orchestra.run.smithi116.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 5.4 MB/s | 5.1 MB 00:00 2024-03-29T10:49:31.031 INFO:teuthology.orchestra.run.smithi116.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 543 kB/s | 45 kB 00:00 2024-03-29T10:49:31.382 INFO:teuthology.orchestra.run.smithi045.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.4 MB/s | 3.1 MB 00:00 2024-03-29T10:49:31.640 INFO:teuthology.orchestra.run.smithi116.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 216 kB/s | 131 kB 00:00 2024-03-29T10:49:31.707 INFO:teuthology.orchestra.run.smithi116.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-29T10:49:31.841 INFO:teuthology.orchestra.run.smithi116.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.3 MB/s | 320 kB 00:00 2024-03-29T10:49:31.974 INFO:teuthology.orchestra.run.smithi116.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 298 kB 00:00 2024-03-29T10:49:32.058 INFO:teuthology.orchestra.run.smithi116.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T10:49:32.166 INFO:teuthology.orchestra.run.smithi116.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 976 kB/s | 106 kB 00:00 2024-03-29T10:49:32.216 INFO:teuthology.orchestra.run.smithi116.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T10:49:32.250 INFO:teuthology.orchestra.run.smithi116.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T10:49:32.309 INFO:teuthology.orchestra.run.smithi116.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T10:49:32.351 INFO:teuthology.orchestra.run.smithi116.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-29T10:49:32.384 INFO:teuthology.orchestra.run.smithi116.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T10:49:32.426 INFO:teuthology.orchestra.run.smithi116.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:49:32.535 INFO:teuthology.orchestra.run.smithi116.stdout:(19/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.1 MB/s | 9.7 MB 00:04 2024-03-29T10:49:32.560 INFO:teuthology.orchestra.run.smithi116.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 260 kB/s | 35 kB 00:00 2024-03-29T10:49:32.585 INFO:teuthology.orchestra.run.smithi116.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 930 kB/s | 46 kB 00:00 2024-03-29T10:49:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-29T10:49:32.828 INFO:teuthology.orchestra.run.smithi116.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-29T10:49:32.878 INFO:teuthology.orchestra.run.smithi116.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-03-29T10:49:32.907 INFO:teuthology.orchestra.run.smithi045.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 3.4 MB 00:02 2024-03-29T10:49:32.911 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:32.912 INFO:teuthology.orchestra.run.smithi045.stdout:Total 7.3 MB/s | 58 MB 00:07 2024-03-29T10:49:32.912 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:49:32.929 INFO:teuthology.orchestra.run.smithi116.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2024-03-29T10:49:32.954 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:49:32.955 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:49:32.995 INFO:teuthology.orchestra.run.smithi116.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T10:49:33.080 INFO:teuthology.orchestra.run.smithi116.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-29T10:49:33.104 INFO:teuthology.orchestra.run.smithi116.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-29T10:49:33.205 INFO:teuthology.orchestra.run.smithi116.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-29T10:49:33.446 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:49:33.446 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:49:33.822 INFO:teuthology.orchestra.run.smithi116.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.7 MB/s | 3.4 MB 00:00 2024-03-29T10:49:34.022 INFO:teuthology.orchestra.run.smithi116.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.3 MB/s | 20 MB 00:06 2024-03-29T10:49:34.130 INFO:teuthology.orchestra.run.smithi116.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.4 MB/s | 3.1 MB 00:00 2024-03-29T10:49:34.135 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:34.136 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.8 MB/s | 58 MB 00:06 2024-03-29T10:49:34.136 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:49:34.174 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:49:34.174 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:49:34.633 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:49:34.634 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:49:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:49:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:49:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:49:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:49:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:49:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:49:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:49:36.117 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:49:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:49:36.182 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:49:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:36.252 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:36.284 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:49:36.333 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:36.344 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:49:36.367 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:49:36.367 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:36.384 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:49:36.518 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:49:36.525 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:49:36.580 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:49:36.620 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:49:36.653 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:49:36.984 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:49:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:49:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:37.021 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:37.106 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:37.112 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:37.129 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:49:37.169 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:49:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:49:37.230 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:37.293 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:37.322 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:49:37.366 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:49:37.434 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:37.458 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:37.504 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:37.539 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:49:37.645 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:49:37.649 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:49:37.677 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:37.682 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:49:37.723 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:49:37.731 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:49:37.793 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:49:37.803 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:49:37.825 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:49:37.829 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:49:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:49:38.180 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:38.253 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:38.285 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:49:38.328 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:49:38.787 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:49:38.816 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:38.955 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:49:38.984 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:49:40.020 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:40.644 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:40.679 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:40.758 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:40.758 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:49:40.758 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:49:40.758 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:40.797 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:41.194 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:41.239 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:41.794 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:41.960 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:49:41.960 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:49:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:41.986 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:50.829 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:50.829 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-03-29T10:49:50.830 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:51.254 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-29T10:49:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-29T10:49:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-29T10:49:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-29T10:49:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-29T10:49:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-29T10:49:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-29T10:49:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:51.753 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:51.778 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:51.778 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:51.778 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:49:51.778 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:51.778 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:51.778 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:51.780 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:51.843 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:51.844 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:52.130 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:52.155 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:52.156 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:52.156 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:49:52.156 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:52.156 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:52.156 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:52.157 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:52.241 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:52.241 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:49:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:49:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:49:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:49:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:49:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:49:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:49:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:49:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:49:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:49:53.388 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:49:53.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:49:53.390 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:49:53.390 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:49:53.390 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:49:53.390 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:49:53.390 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:49:53.391 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:49:53.391 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:49:53.391 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:49:53.391 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:53.391 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:49:53.391 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.120 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:49:54.121 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:49:54.122 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:54.123 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:49:54.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-03-29T10:49:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:54.465 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-29T10:49:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.465 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:49:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:54.468 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:49:54.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2024-03-29T10:49:55.034 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:33 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:49:55.192 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:49:55.193 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:49:55.194 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:49:55.194 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:55.194 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:49:55.194 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:49:55.194 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-03-29T10:49:55.194 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:49:55.195 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 48 M 2024-03-29T10:49:55.195 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 204 M 2024-03-29T10:49:55.195 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:49:55.336 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:31 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:49:55.406 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T10:49:55.457 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:49:55.482 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.6 MB/s | 63 kB 00:00 2024-03-29T10:49:55.486 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:55.487 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:49:55.488 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:49:55.488 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-03-29T10:49:55.488 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:49:55.488 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 48 M 2024-03-29T10:49:55.488 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 204 M 2024-03-29T10:49:55.488 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:49:55.762 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:49:55.812 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:49:55.846 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-29T10:49:55.949 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 235 kB/s | 163 kB 00:00 2024-03-29T10:49:55.991 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-03-29T10:49:56.071 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 832 kB/s | 187 kB 00:00 2024-03-29T10:49:56.263 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 247 kB/s | 163 kB 00:00 2024-03-29T10:49:58.716 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 14 MB/s | 47 MB 00:03 2024-03-29T10:49:58.719 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:58.720 INFO:teuthology.orchestra.run.smithi045.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-29T10:49:58.735 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:49:58.758 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:49:58.758 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:49:59.157 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:49:59.158 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:49:59.746 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:49:59.826 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:49:59.864 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:49:59.904 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:59.954 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:49:59.963 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 47 MB 00:04 2024-03-29T10:49:59.966 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:59.966 INFO:teuthology.orchestra.run.smithi116.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-29T10:49:59.978 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:59.981 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:00.002 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:00.002 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:00.388 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:00.388 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:00.972 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:01.046 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:50:01.088 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:50:01.128 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:50:01.172 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:50:01.199 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:50:04.698 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:50:04.772 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:50:05.670 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:50:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:50:07.196 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:50:07.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:50:07.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:50:07.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:50:07.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:50:07.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:07.559 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:07.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-03-29T10:50:08.138 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:50:08.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:50:08.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:50:08.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:50:08.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:50:08.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:50:08.433 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:46 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:08.487 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:08.597 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:50:08.600 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:50:08.601 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:50:08.602 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout:Install 36 Packages 2024-03-29T10:50:08.603 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:08.604 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 30 M 2024-03-29T10:50:08.604 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 107 M 2024-03-29T10:50:08.604 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:08.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2024-03-29T10:50:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:45 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:09.413 INFO:teuthology.orchestra.run.smithi045.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 24 kB/s | 6.6 kB 00:00 2024-03-29T10:50:09.502 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================= 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================= 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:50:09.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:50:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================= 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout:Install 36 Packages 2024-03-29T10:50:09.512 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:09.513 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 30 M 2024-03-29T10:50:09.513 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 107 M 2024-03-29T10:50:09.513 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:09.538 INFO:teuthology.orchestra.run.smithi045.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-03-29T10:50:09.589 INFO:teuthology.orchestra.run.smithi045.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.7 MB/s | 2.1 MB 00:00 2024-03-29T10:50:09.672 INFO:teuthology.orchestra.run.smithi045.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.9 MB/s | 247 kB 00:00 2024-03-29T10:50:09.798 INFO:teuthology.orchestra.run.smithi045.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:50:09.840 INFO:teuthology.orchestra.run.smithi045.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:50:09.865 INFO:teuthology.orchestra.run.smithi045.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:50:09.890 INFO:teuthology.orchestra.run.smithi045.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T10:50:09.916 INFO:teuthology.orchestra.run.smithi045.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T10:50:09.958 INFO:teuthology.orchestra.run.smithi045.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:50:10.024 INFO:teuthology.orchestra.run.smithi045.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:50:10.058 INFO:teuthology.orchestra.run.smithi045.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:50:10.092 INFO:teuthology.orchestra.run.smithi045.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:50:10.117 INFO:teuthology.orchestra.run.smithi045.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 7.3 MB/s | 187 kB 00:00 2024-03-29T10:50:10.259 INFO:teuthology.orchestra.run.smithi045.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-03-29T10:50:10.379 INFO:teuthology.orchestra.run.smithi116.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-03-29T10:50:10.443 INFO:teuthology.orchestra.run.smithi045.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 17 MB 00:00 2024-03-29T10:50:10.460 INFO:teuthology.orchestra.run.smithi045.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-29T10:50:10.529 INFO:teuthology.orchestra.run.smithi116.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.4 MB/s | 2.1 MB 00:00 2024-03-29T10:50:10.552 INFO:teuthology.orchestra.run.smithi045.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 475 kB/s | 43 kB 00:00 2024-03-29T10:50:10.602 INFO:teuthology.orchestra.run.smithi045.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 952 kB/s | 151 kB 00:00 2024-03-29T10:50:10.644 INFO:teuthology.orchestra.run.smithi045.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T10:50:10.686 INFO:teuthology.orchestra.run.smithi045.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-29T10:50:10.728 INFO:teuthology.orchestra.run.smithi045.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T10:50:10.753 INFO:teuthology.orchestra.run.smithi045.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-29T10:50:10.778 INFO:teuthology.orchestra.run.smithi045.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-29T10:50:10.804 INFO:teuthology.orchestra.run.smithi045.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-29T10:50:10.829 INFO:teuthology.orchestra.run.smithi045.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-03-29T10:50:10.854 INFO:teuthology.orchestra.run.smithi045.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-03-29T10:50:10.872 INFO:teuthology.orchestra.run.smithi045.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-29T10:50:10.914 INFO:teuthology.orchestra.run.smithi045.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-03-29T10:50:10.929 INFO:teuthology.orchestra.run.smithi116.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 1.5 MB 00:00 2024-03-29T10:50:10.939 INFO:teuthology.orchestra.run.smithi045.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T10:50:10.964 INFO:teuthology.orchestra.run.smithi045.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-29T10:50:10.981 INFO:teuthology.orchestra.run.smithi045.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-03-29T10:50:11.015 INFO:teuthology.orchestra.run.smithi045.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T10:50:11.040 INFO:teuthology.orchestra.run.smithi045.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-29T10:50:11.057 INFO:teuthology.orchestra.run.smithi045.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-03-29T10:50:11.088 INFO:teuthology.orchestra.run.smithi116.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.5 MB/s | 247 kB 00:00 2024-03-29T10:50:11.272 INFO:teuthology.orchestra.run.smithi116.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T10:50:11.439 INFO:teuthology.orchestra.run.smithi116.stdout:(6/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 17 MB 00:00 2024-03-29T10:50:11.455 INFO:teuthology.orchestra.run.smithi116.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 786 kB/s | 144 kB 00:00 2024-03-29T10:50:11.497 INFO:teuthology.orchestra.run.smithi116.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 544 kB/s | 32 kB 00:00 2024-03-29T10:50:11.523 INFO:teuthology.orchestra.run.smithi116.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 706 kB/s | 47 kB 00:00 2024-03-29T10:50:11.548 INFO:teuthology.orchestra.run.smithi116.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-29T10:50:11.615 INFO:teuthology.orchestra.run.smithi116.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-29T10:50:11.641 INFO:teuthology.orchestra.run.smithi045.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 4.7 MB 00:02 2024-03-29T10:50:11.644 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:11.644 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-03-29T10:50:11.669 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:11.716 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:11.716 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:11.790 INFO:teuthology.orchestra.run.smithi116.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 4.7 MB/s | 1.1 MB 00:00 2024-03-29T10:50:11.816 INFO:teuthology.orchestra.run.smithi116.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 514 kB/s | 103 kB 00:00 2024-03-29T10:50:11.857 INFO:teuthology.orchestra.run.smithi116.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-03-29T10:50:11.883 INFO:teuthology.orchestra.run.smithi116.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-29T10:50:12.131 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:12.132 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:12.292 INFO:teuthology.orchestra.run.smithi116.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 288 kB/s | 125 kB 00:00 2024-03-29T10:50:12.425 INFO:teuthology.orchestra.run.smithi116.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-29T10:50:12.467 INFO:teuthology.orchestra.run.smithi116.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-29T10:50:12.517 INFO:teuthology.orchestra.run.smithi116.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-29T10:50:12.584 INFO:teuthology.orchestra.run.smithi116.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-29T10:50:12.618 INFO:teuthology.orchestra.run.smithi116.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 321 kB/s | 11 kB 00:00 2024-03-29T10:50:12.635 INFO:teuthology.orchestra.run.smithi116.stdout:(22/36): lua-devel-5.4.4-4.el9.x86_64.rpm 30 kB/s | 22 kB 00:00 2024-03-29T10:50:12.660 INFO:teuthology.orchestra.run.smithi116.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 430 kB/s | 18 kB 00:00 2024-03-29T10:50:12.694 INFO:teuthology.orchestra.run.smithi116.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 587 kB/s | 19 kB 00:00 2024-03-29T10:50:12.719 INFO:teuthology.orchestra.run.smithi116.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 276 kB/s | 23 kB 00:00 2024-03-29T10:50:12.761 INFO:teuthology.orchestra.run.smithi116.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-29T10:50:12.786 INFO:teuthology.orchestra.run.smithi116.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 692 kB/s | 46 kB 00:00 2024-03-29T10:50:12.811 INFO:teuthology.orchestra.run.smithi116.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T10:50:12.853 INFO:teuthology.orchestra.run.smithi116.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2024-03-29T10:50:12.887 INFO:teuthology.orchestra.run.smithi116.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-29T10:50:12.904 INFO:teuthology.orchestra.run.smithi116.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T10:50:12.929 INFO:teuthology.orchestra.run.smithi116.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 856 kB/s | 36 kB 00:00 2024-03-29T10:50:12.954 INFO:teuthology.orchestra.run.smithi116.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T10:50:12.996 INFO:teuthology.orchestra.run.smithi116.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-03-29T10:50:13.022 INFO:teuthology.orchestra.run.smithi116.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-03-29T10:50:13.313 INFO:teuthology.orchestra.run.smithi116.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.6 MB/s | 4.7 MB 00:02 2024-03-29T10:50:13.318 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:13.318 INFO:teuthology.orchestra.run.smithi116.stdout:Total 7.8 MB/s | 30 MB 00:03 2024-03-29T10:50:13.342 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:13.386 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:13.387 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:13.593 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:50:13.630 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:50:13.663 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:50:13.729 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:50:13.770 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:50:13.803 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:13.803 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:13.803 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:50:13.899 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:50:13.976 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:50:14.023 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:50:14.098 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:50:14.149 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:50:14.187 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:50:14.246 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:50:14.318 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:50:14.514 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:50:14.566 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:50:14.605 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:50:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:50:14.678 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:50:14.757 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:50:14.788 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:50:14.822 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:50:14.862 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:50:14.906 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:50:14.949 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:50:15.051 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:50:15.088 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:50:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:15.164 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:50:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:50:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:50:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:50:15.315 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:50:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:50:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:50:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:50:15.445 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:50:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:50:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:50:15.612 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:50:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:50:15.651 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:50:15.676 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:50:15.676 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:15.676 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:50:15.676 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:50:15.676 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:50:15.676 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:50:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:50:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:50:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:50:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:50:16.104 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:50:16.146 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:50:16.185 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:50:16.219 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:50:16.309 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:50:16.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:50:16.417 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:50:16.452 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:50:16.486 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:50:16.576 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:50:16.617 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:50:16.689 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:50:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:50:16.963 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:50:17.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:50:17.155 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:50:17.181 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:50:17.181 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:17.181 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:50:17.181 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:50:17.181 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:50:17.181 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:50:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:50:17.211 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:17.211 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:50:17.211 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:50:17.211 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:50:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:17.669 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:50:17.692 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:50:17.692 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:17.692 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:50:17.692 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:50:17.692 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:50:17.692 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:17.903 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:50:17.926 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:50:17.926 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:17.927 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:50:17.927 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:50:17.927 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:50:17.927 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:50:18.631 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:50:18.657 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:50:18.658 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:18.658 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:50:18.658 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:50:18.658 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:50:18.658 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:50:19.122 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:50:19.122 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:19.122 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:50:19.122 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:50:19.122 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:50:19.122 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:19.324 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:50:19.347 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:50:19.348 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:19.348 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:50:19.348 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:50:19.348 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:50:19.348 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:19.406 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:50:19.616 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:50:19.617 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:50:19.620 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:20.478 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:50:20.479 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:50:20.480 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:20.481 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:20.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:50:21.114 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:50:21.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:50:21.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:50:21.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:50:21.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:50:21.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:50:21.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:50:21.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:50:21.403 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:21.505 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:21.560 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:21.562 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:50:21.562 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:21.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-03-29T10:50:21.999 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:50:21.999 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:21.999 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:50:22.001 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:22.002 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:22.209 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:22.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2024-03-29T10:50:22.364 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:22.365 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:22.365 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:22.365 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:22.365 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:22.365 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:50:22.365 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 747 k 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 754 k 2024-03-29T10:50:22.366 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:22.794 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T10:50:22.795 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:22.795 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T10:50:22.795 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:22.800 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:22.801 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:22.807 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:22.807 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:22.854 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:22.885 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:22.951 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:23.002 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:23.003 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:50:23.003 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:23.075 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2024-03-29T10:50:23.302 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:23.393 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:23.636 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:23.765 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:23.787 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:23.788 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:50:23.789 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:23.789 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 747 k 2024-03-29T10:50:23.789 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 754 k 2024-03-29T10:50:23.789 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:24.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:24.056 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:24.056 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:24.056 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:24.056 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:24.056 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:24.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:50:24.356 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.3 MB/s | 747 kB 00:00 2024-03-29T10:50:24.357 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:24.357 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.3 MB/s | 747 kB 00:00 2024-03-29T10:50:24.357 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:24.362 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:24.363 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:24.368 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:24.368 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:24.442 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:24.844 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:24.919 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:25.001 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:25.002 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:25.002 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T10:50:25.002 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:25.002 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:25.002 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 145 k 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 436 k 2024-03-29T10:50:25.003 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:25.270 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 486 kB/s | 145 kB 00:00 2024-03-29T10:50:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:25.302 INFO:teuthology.orchestra.run.smithi045.stdout:Total 484 kB/s | 145 kB 00:00 2024-03-29T10:50:25.303 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:25.311 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:25.311 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:25.366 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:25.366 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:25.508 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:25.551 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:25.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:25.552 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:25.552 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:25.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:25.552 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:25.561 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:25.586 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:25.586 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:25.586 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:50:25.586 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:25.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:50:26.318 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:26.467 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:26.468 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 145 k 2024-03-29T10:50:26.469 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 436 k 2024-03-29T10:50:26.469 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:26.572 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:26.572 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:26.572 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:26.572 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:26.572 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:26.572 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:26.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-03-29T10:50:26.862 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 370 kB/s | 145 kB 00:00 2024-03-29T10:50:26.863 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:26.864 INFO:teuthology.orchestra.run.smithi116.stdout:Total 369 kB/s | 145 kB 00:00 2024-03-29T10:50:26.864 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:26.870 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:26.871 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:26.926 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:26.926 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:27.121 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:27.145 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:27.145 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:50:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:27.426 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:27.528 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:27.584 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:27.585 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:50:27.585 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:27.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:50:28.142 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:28.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:28.147 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:28.147 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:28.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:28.148 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:28.225 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:28.384 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:50:28.385 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-03-29T10:50:28.386 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:28.387 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.9 M 2024-03-29T10:50:28.387 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-03-29T10:50:28.387 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:28.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2024-03-29T10:50:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:29.103 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:29.141 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 36 kB/s | 15 kB 00:00 2024-03-29T10:50:29.155 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:29.156 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:50:29.156 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:29.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:50:29.249 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-29T10:50:29.408 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-29T10:50:29.433 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-29T10:50:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-29T10:50:29.784 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-29T10:50:29.787 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:29.826 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.6 MB/s | 1.8 MB 00:01 2024-03-29T10:50:29.937 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:29.938 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================== 2024-03-29T10:50:29.938 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:29.938 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================== 2024-03-29T10:50:29.938 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:29.938 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:50:29.938 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================== 2024-03-29T10:50:29.939 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-03-29T10:50:29.940 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:29.940 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 6.9 M 2024-03-29T10:50:29.940 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-03-29T10:50:29.940 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:30.118 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 495 kB/s | 144 kB 00:00 2024-03-29T10:50:30.176 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 532 kB/s | 31 kB 00:00 2024-03-29T10:50:30.385 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 907 kB/s | 188 kB 00:00 2024-03-29T10:50:30.427 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.2 MB/s | 2.0 MB 00:00 2024-03-29T10:50:30.619 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 713 kB/s | 863 kB 00:01 2024-03-29T10:50:30.623 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:30.623 INFO:teuthology.orchestra.run.smithi045.stdout:Total 3.1 MB/s | 6.9 MB 00:02 2024-03-29T10:50:30.643 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:30.664 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:30.664 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:30.807 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 39 kB/s | 15 kB 00:00 2024-03-29T10:50:30.824 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 77 kB/s | 31 kB 00:00 2024-03-29T10:50:30.850 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:30.850 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:30.966 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T10:50:31.008 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T10:50:31.233 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.4 MB/s | 548 kB 00:00 2024-03-29T10:50:31.267 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 583 kB/s | 19 kB 00:00 2024-03-29T10:50:31.292 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-03-29T10:50:31.362 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:31.418 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-29T10:50:31.445 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:50:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 619 kB/s | 31 kB 00:00 2024-03-29T10:50:31.488 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:50:31.535 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:50:31.543 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-29T10:50:31.573 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:50:31.643 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.5 MB/s | 1.8 MB 00:01 2024-03-29T10:50:31.678 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:50:31.685 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-03-29T10:50:31.688 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:31.689 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-03-29T10:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:31.723 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:31.723 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:31.811 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:50:31.903 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:50:31.922 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:31.923 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:31.946 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:50:31.974 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:31.997 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:50:32.242 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:50:32.272 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:32.428 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:32.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:50:32.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:50:32.584 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:50:32.623 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:50:32.725 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:50:32.848 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:50:32.951 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:50:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:50:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:50:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:50:33.042 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:50:33.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:50:33.044 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:50:33.072 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:33.104 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:50:33.347 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:50:33.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:33.416 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:50:33.417 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:50:33.418 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:33.418 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:33.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:50:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:50:34.229 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:34.387 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:34.388 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:50:34.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-03-29T10:50:34.390 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:34.391 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2024-03-29T10:50:34.391 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 209 M 2024-03-29T10:50:34.391 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:50:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:34.565 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:34.566 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:34.718 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-03-29T10:50:34.752 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-03-29T10:50:34.794 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 34 kB 00:00 2024-03-29T10:50:34.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:50:34.844 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:50:35.044 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T10:50:35.106 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T10:50:35.145 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:50:35.312 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-29T10:50:35.352 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:35.470 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T10:50:35.496 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T10:50:35.504 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:35.505 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-29T10:50:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:35.505 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-29T10:50:35.505 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:50:35.505 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:35.506 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-29T10:50:35.507 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-03-29T10:50:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:35.507 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 37 M 2024-03-29T10:50:35.507 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 209 M 2024-03-29T10:50:35.507 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:35.621 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-29T10:50:35.791 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 253 kB/s | 32 kB 00:00 2024-03-29T10:50:35.813 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.9 MB/s | 7.4 MB 00:01 2024-03-29T10:50:35.815 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:35.815 INFO:teuthology.orchestra.run.smithi045.stdout:Total 26 MB/s | 37 MB 00:01 2024-03-29T10:50:35.816 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 691 kB/s | 17 kB 00:00 2024-03-29T10:50:35.856 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:35.867 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-03-29T10:50:35.880 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:35.880 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:36.034 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T10:50:36.134 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.4 MB/s | 3.0 MB 00:00 2024-03-29T10:50:36.218 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-29T10:50:36.234 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:50:36.256 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:36.256 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:36.535 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-03-29T10:50:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:37.035 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 391 kB/s | 195 kB 00:00 2024-03-29T10:50:37.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:50:37.092 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:50:37.195 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:37.202 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-1 4.8 MB/s | 7.4 MB 00:01 2024-03-29T10:50:37.227 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 810 kB/s | 803 kB 00:00 2024-03-29T10:50:37.229 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:50:37.413 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:37.436 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:37.513 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:50:37.994 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 256 kB/s | 245 kB 00:00 2024-03-29T10:50:37.997 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:37.997 INFO:teuthology.orchestra.run.smithi116.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-29T10:50:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:38.066 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:38.066 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:38.207 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:50:38.322 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:50:38.427 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:38.427 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:39.124 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:39.200 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:50:39.240 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:50:39.335 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:39.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:50:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:39.583 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:50:39.660 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:50:40.320 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:50:40.430 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:50:40.731 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:50:40.754 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:41.688 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:50:42.706 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:50:42.733 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:43.252 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:43.252 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:50:43.252 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:50:43.252 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:43.252 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:50:43.253 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:50:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:50:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:43.641 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:50:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:43.642 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:44.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:50:44.586 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:50:44.746 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-03-29T10:50:44.747 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:44.748 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2024-03-29T10:50:44.748 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-03-29T10:50:44.748 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:45.008 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 187 kB/s | 24 kB 00:00 2024-03-29T10:50:45.033 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-29T10:50:45.075 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T10:50:45.100 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T10:50:45.126 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:50:45.154 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T10:50:45.211 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:45.212 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:50:45.213 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:50:45.213 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:50:45.213 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:50:45.213 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:50:45.218 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-03-29T10:50:45.243 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-03-29T10:50:45.352 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 110 kB/s | 52 kB 00:00 2024-03-29T10:50:45.377 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-29T10:50:45.419 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-03-29T10:50:45.444 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T10:50:45.469 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-29T10:50:45.471 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:45.471 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T10:50:45.472 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:45.486 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:45.486 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:45.582 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:45.582 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:45.636 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:50:45.636 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:45.636 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:45.636 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:45.636 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:45.637 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:46.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:50:46.111 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:46.299 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:50:46.356 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:50:46.412 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:50:46.449 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:50:46.492 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:50:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:50:46.602 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:46.700 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:50:46.751 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:50:46.753 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:46.754 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2024-03-29T10:50:46.754 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:46.754 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2024-03-29T10:50:46.754 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:46.754 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:50:46.754 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:50:46.755 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.0 M 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-03-29T10:50:46.756 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:46.976 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 143 kB/s | 24 kB 00:00 2024-03-29T10:50:47.001 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 92 kB/s | 18 kB 00:00 2024-03-29T10:50:47.109 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 174 kB 00:00 2024-03-29T10:50:47.135 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 996 kB/s | 132 kB 00:00 2024-03-29T10:50:47.185 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.8 MB/s | 211 kB 00:00 2024-03-29T10:50:47.210 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-29T10:50:47.403 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 87 kB/s | 52 kB 00:00 2024-03-29T10:50:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:50:47.527 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 94 kB/s | 32 kB 00:00 2024-03-29T10:50:47.553 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2024-03-29T10:50:47.564 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:50:47.598 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:50:47.648 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:50:47.676 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:47.703 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 396 kB/s | 59 kB 00:00 2024-03-29T10:50:47.795 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 517 kB/s | 203 kB 00:00 2024-03-29T10:50:47.820 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 769 kB/s | 90 kB 00:00 2024-03-29T10:50:47.853 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T10:50:47.857 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:47.858 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-29T10:50:47.858 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:47.871 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:47.871 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:47.961 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:47.961 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:48.490 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:48.531 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:48.531 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:50:48.531 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:50:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:50:48.535 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:50:48.535 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:50:48.535 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:50:48.671 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:50:48.733 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:50:48.790 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:50:48.822 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:50:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:50:48.889 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:50:48.889 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:48.889 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:48.889 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:48.889 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:50:48.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:50:48.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:50:48.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:50:48.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:48.891 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:48.961 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:50:49.068 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:50:49.118 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:50:49.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:50:49.727 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:49.835 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:50:49.885 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:49.886 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:50:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.8 M 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 33 M 2024-03-29T10:50:49.888 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:50:49.921 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:50:49.969 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:50:49.991 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.3 MB/s | 248 kB 00:00 2024-03-29T10:50:50.325 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:50:50.492 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-29T10:50:50.518 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 304 kB/s | 58 kB 00:00 2024-03-29T10:50:50.542 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:50:50.768 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:50:50.834 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 175 kB/s | 138 kB 00:00 2024-03-29T10:50:50.837 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:50.837 INFO:teuthology.orchestra.run.smithi045.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-03-29T10:50:50.851 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:50.862 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:50.862 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:50.956 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:50.957 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:51.143 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:50:51.143 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:51.143 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:51.143 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:50:51.144 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:50:51.145 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:50:51.145 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:50:51.145 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:50:51.145 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:50:51.145 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:50:51.145 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:51.146 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:51.251 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:51.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:50:51.423 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:50:51.487 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:50:52.126 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:50:52.127 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 6.8 M 2024-03-29T10:50:52.128 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2024-03-29T10:50:52.129 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:52.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:50:52.202 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:50:52.231 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T10:50:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 284 kB/s | 138 kB 00:00 2024-03-29T10:50:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 578 kB/s | 58 kB 00:00 2024-03-29T10:50:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.6 MB/s | 548 kB 00:00 2024-03-29T10:50:52.964 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T10:50:53.018 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:53.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:50:53.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:50:53.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:50:53.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:50:53.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:50:53.329 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:50:53.329 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:53.329 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:53.329 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:53.329 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:50:53.330 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:50:53.330 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:50:53.330 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:50:53.330 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:50:53.330 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:53.330 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:53.334 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 247 kB/s | 248 kB 00:01 2024-03-29T10:50:53.334 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:53.334 INFO:teuthology.orchestra.run.smithi116.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-03-29T10:50:53.349 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:53.361 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:53.361 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:53.451 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:53.452 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:53.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-03-29T10:50:53.745 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:50:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:50:54.131 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:54.288 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:54.288 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:54.288 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:54.288 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-03-29T10:50:54.289 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:54.290 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 964 k 2024-03-29T10:50:54.290 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.8 M 2024-03-29T10:50:54.290 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:54.411 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 649 kB/s | 79 kB 00:00 2024-03-29T10:50:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:50:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:50:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:50:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:55.328 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 853 kB/s | 885 kB 00:01 2024-03-29T10:50:55.329 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:55.330 INFO:teuthology.orchestra.run.smithi045.stdout:Total 926 kB/s | 964 kB 00:01 2024-03-29T10:50:55.330 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:55.354 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:55.354 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:55.421 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:55.421 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:50:55.421 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:50:55.421 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:50:55.421 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:50:55.421 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:50:55.435 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:55.436 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:55.721 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:50:55.721 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:55.722 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:55.766 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:50:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:55.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2024-03-29T10:50:56.532 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:50:56.555 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:56.555 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:50:56.682 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:50:56.682 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:56.682 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 964 k 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 2.8 M 2024-03-29T10:50:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:50:56.801 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 670 kB/s | 79 kB 00:00 2024-03-29T10:50:56.910 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:50:56.910 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:56.910 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:50:56.910 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:50:56.911 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:56.911 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:50:57.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-volume 2024-03-29T10:50:57.734 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 842 kB/s | 885 kB 00:01 2024-03-29T10:50:57.736 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:57.736 INFO:teuthology.orchestra.run.smithi116.stdout:Total 915 kB/s | 964 kB 00:01 2024-03-29T10:50:57.736 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:50:57.759 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:50:57.759 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:50:57.776 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:50:57.835 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:50:57.835 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:50:57.933 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:50:57.934 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 332 k 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 1.5 M 2024-03-29T10:50:57.935 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:50:58.020 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:50:58.067 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 523 kB/s | 69 kB 00:00 2024-03-29T10:50:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:50:58.227 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:58.625 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 382 kB/s | 264 kB 00:00 2024-03-29T10:50:58.626 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:58.627 INFO:teuthology.orchestra.run.smithi045.stdout:Total 480 kB/s | 332 kB 00:00 2024-03-29T10:50:58.627 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:50:58.637 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:50:58.637 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:50:58.682 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:50:58.683 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:50:58.836 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:50:58.931 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:58.932 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:50:59.023 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:50:59.061 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:59.077 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:59.077 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:59.077 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:50:59.077 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:50:59.273 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:50:59.273 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:59.273 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:50:59.273 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:50:59.273 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:50:59.274 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:50:59.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-volume 2024-03-29T10:50:59.794 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:00.151 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:00.157 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:00.308 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:00.309 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:00.309 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-29T10:51:00.309 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:00.309 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:51:00.309 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:51:00.309 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 332 k 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 1.5 M 2024-03-29T10:51:00.310 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:51:00.439 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 536 kB/s | 69 kB 00:00 2024-03-29T10:51:00.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-03-29T10:51:00.939 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 420 kB/s | 264 kB 00:00 2024-03-29T10:51:00.940 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:00.940 INFO:teuthology.orchestra.run.smithi116.stdout:Total 527 kB/s | 332 kB 00:00 2024-03-29T10:51:00.940 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:51:00.949 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:51:00.949 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:51:00.988 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:51:00.988 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:51:01.055 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:01.138 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:51:01.213 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:01.214 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T10:51:01.214 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:01.214 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:51:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 127 k 2024-03-29T10:51:01.215 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 456 k 2024-03-29T10:51:01.216 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:51:01.296 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:51:01.325 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:51:01.340 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:51:01.341 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:51:01.341 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:51:01.341 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:01.849 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 199 kB/s | 127 kB 00:00 2024-03-29T10:51:01.850 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:01.850 INFO:teuthology.orchestra.run.smithi045.stdout:Total 199 kB/s | 127 kB 00:00 2024-03-29T10:51:01.850 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:51:01.857 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:51:01.857 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:51:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:51:01.917 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:51:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:51:02.063 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:51:02.176 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:02.379 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:02.588 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:02.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2024-03-29T10:51:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:02.868 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:51:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:02.868 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:03.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-03-29T10:51:03.293 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:03.445 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:51:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:03.447 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:51:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:03.447 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 127 k 2024-03-29T10:51:03.447 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 456 k 2024-03-29T10:51:03.447 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:51:03.695 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:03.797 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:03.853 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:03.854 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:03.854 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:03.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-03-29T10:51:04.003 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 228 kB/s | 127 kB 00:00 2024-03-29T10:51:04.004 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:04.004 INFO:teuthology.orchestra.run.smithi116.stdout:Total 227 kB/s | 127 kB 00:00 2024-03-29T10:51:04.004 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:51:04.009 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:51:04.010 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:51:04.065 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:51:04.065 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:51:04.206 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:51:04.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:04.505 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:04.664 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:04.664 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 31 k 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 140 k 2024-03-29T10:51:04.665 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:51:04.828 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:05.012 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 90 kB/s | 31 kB 00:00 2024-03-29T10:51:05.013 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:05.013 INFO:teuthology.orchestra.run.smithi045.stdout:Total 90 kB/s | 31 kB 00:00 2024-03-29T10:51:05.013 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:51:05.016 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:51:05.016 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:51:05.037 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:51:05.038 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:51:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:05.129 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:51:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:05.129 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:05.161 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:51:05.254 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:05.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2024-03-29T10:51:05.638 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:05.900 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:05.900 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:05.900 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:51:05.900 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:05.900 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:05.900 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:05.929 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:06.029 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:06.082 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:06.083 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:06.083 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:06.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-03-29T10:51:06.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2024-03-29T10:51:06.659 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:06.735 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:06.759 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:06.813 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:06.815 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:06.815 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:06.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-03-29T10:51:06.888 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 31 k 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 140 k 2024-03-29T10:51:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:51:07.223 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 94 kB/s | 31 kB 00:00 2024-03-29T10:51:07.223 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:07.223 INFO:teuthology.orchestra.run.smithi116.stdout:Total 93 kB/s | 31 kB 00:00 2024-03-29T10:51:07.224 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:51:07.227 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:51:07.227 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:51:07.251 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:51:07.251 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:51:07.395 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:51:07.444 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:07.480 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:07.544 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:07.599 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:07.600 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:07.601 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:07.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-03-29T10:51:07.832 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:08.111 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:08.111 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:08.111 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:51:08.112 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:08.112 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:08.112 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:08.226 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:08.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2024-03-29T10:51:08.329 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:08.384 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:08.385 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:08.385 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:08.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-03-29T10:51:08.848 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:08.944 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:08.996 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:08.997 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:08.997 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:09.010 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:09.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2024-03-29T10:51:09.112 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:09.167 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:09.169 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:09.169 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:09.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-03-29T10:51:09.611 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:09.709 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:09.760 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:09.761 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:09.761 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:09.803 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:09.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2024-03-29T10:51:09.904 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:09.958 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:09.959 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:09.959 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:10.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-03-29T10:51:10.372 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:10.474 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:10.526 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:10.527 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:10.528 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:10.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2024-03-29T10:51:10.595 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:10.696 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:10.751 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:10.752 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T10:51:10.752 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:10.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-03-29T10:51:11.144 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:11.242 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:11.295 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:11.295 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:11.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2024-03-29T10:51:11.377 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:11.537 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:11.538 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 85 k 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 231 k 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:51:11.912 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:11.983 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 192 kB/s | 85 kB 00:00 2024-03-29T10:51:11.983 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:11.984 INFO:teuthology.orchestra.run.smithi045.stdout:Total 191 kB/s | 85 kB 00:00 2024-03-29T10:51:11.984 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:51:11.992 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:51:11.992 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:51:12.010 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:12.049 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:51:12.050 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:51:12.061 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:12.062 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:12.062 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:12.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2024-03-29T10:51:12.199 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:51:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:12.674 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:12.690 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:12.773 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:12.825 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:12.826 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-29T10:51:12.827 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:12.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2024-03-29T10:51:12.991 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:12.991 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:12.992 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:51:12.992 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:12.992 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:12.992 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:13.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-03-29T10:51:13.457 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:13.616 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 85 k 2024-03-29T10:51:13.617 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 231 k 2024-03-29T10:51:13.617 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:51:13.823 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:13.951 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 255 kB/s | 85 kB 00:00 2024-03-29T10:51:13.952 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:13.952 INFO:teuthology.orchestra.run.smithi116.stdout:Total 253 kB/s | 85 kB 00:00 2024-03-29T10:51:13.953 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:51:13.960 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:51:13.960 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:51:13.979 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:51:13.980 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:13.981 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:51:13.981 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:13.981 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2024-03-29T10:51:13.981 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 13 M 2024-03-29T10:51:13.981 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:51:14.017 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:51:14.017 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:51:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:51:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:14.944 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:51:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:14.944 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:15.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-03-29T10:51:15.486 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.1 MB/s | 3.1 MB 00:01 2024-03-29T10:51:15.487 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:15.488 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.0 MB/s | 3.1 MB 00:01 2024-03-29T10:51:15.488 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:51:15.500 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:51:15.500 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:51:15.580 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:51:15.580 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:51:15.752 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:15.910 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:15.911 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:51:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:15.912 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.1 M 2024-03-29T10:51:15.912 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 13 M 2024-03-29T10:51:15.912 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:51:16.038 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:51:16.096 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:16.121 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:16.121 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:51:16.121 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:51:16.121 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:51:16.121 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:51:16.121 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:17.225 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:51:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:17.225 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:17.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-03-29T10:51:17.513 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.9 MB/s | 3.1 MB 00:01 2024-03-29T10:51:17.513 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:17.514 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.9 MB/s | 3.1 MB 00:01 2024-03-29T10:51:17.514 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:51:17.525 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:51:17.525 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:51:17.601 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:51:17.601 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:51:18.038 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:51:18.046 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:49:22 AM UTC. 2024-03-29T10:51:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:18.113 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:18.113 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:51:18.114 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:51:18.114 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:51:18.114 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:51:18.114 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:18.201 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:18.203 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T10:51:18.204 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T10:51:18.204 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T10:51:18.204 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:18.204 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 171 k 2024-03-29T10:51:18.204 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 494 k 2024-03-29T10:51:18.204 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T10:51:18.617 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 414 kB/s | 171 kB 00:00 2024-03-29T10:51:18.617 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:18.618 INFO:teuthology.orchestra.run.smithi045.stdout:Total 413 kB/s | 171 kB 00:00 2024-03-29T10:51:18.618 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T10:51:18.626 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T10:51:18.626 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T10:51:18.684 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T10:51:18.684 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T10:51:18.834 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T10:51:18.934 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:19.211 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:19.211 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:19.211 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:51:19.211 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:19.211 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:19.211 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:19.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2024-03-29T10:51:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:19.649 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T10:51:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:19.649 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T10:51:19.884 DEBUG:teuthology.parallel:result is None 2024-03-29T10:51:20.057 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:49:24 AM UTC. 2024-03-29T10:51:20.208 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-29T10:51:20.208 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 171 k 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 494 k 2024-03-29T10:51:20.209 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-29T10:51:20.458 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 691 kB/s | 171 kB 00:00 2024-03-29T10:51:20.458 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:20.459 INFO:teuthology.orchestra.run.smithi116.stdout:Total 686 kB/s | 171 kB 00:00 2024-03-29T10:51:20.459 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-29T10:51:20.466 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-29T10:51:20.466 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-29T10:51:20.522 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-29T10:51:20.523 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-29T10:51:20.672 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-29T10:51:20.782 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:21.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:21.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:21.535 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-29T10:51:21.535 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:21.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:21.536 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-29T10:51:21.813 DEBUG:teuthology.parallel:result is None 2024-03-29T10:51:21.814 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:51:22.115 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:51:22.139 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:22.140 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:22.140 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:51:22.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:51:22.281 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:51:22.304 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:22.304 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:22.304 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:51:22.306 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:51:22.306 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:22.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:51:22.339 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:22.339 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:51:22.374 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:51:22.374 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:22.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:51:22.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:51:22.482 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:22.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:51:22.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:51:22.579 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:51:22.580 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:22.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:51:22.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:51:22.676 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:22.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:51:22.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:51:22.771 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:51:22.771 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:22.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:51:22.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:51:22.866 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:22.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:51:22.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:51:22.961 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:51:23.055 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:51:23.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-03-29T10:51:23.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-03-29T10:51:23.084 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:51:23.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:51:23.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:51:23.152 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:51:23.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:51:23.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:51:23.231 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:51:23.231 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:51:23.232 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622660', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI4p6ibs7lJ1PiB5ycWMubtAd2eIXTbxBFYTi66d687eW0XvLop/s7OH1CsTcIyd817+s3Z3oSShrHyRMeRBVPs=', 'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBVO7ruRl0xSWBFQuwtmzxUOeWSAOQUBI3q2WwjDZWo+Z/fgFJpv8+BTJQkRGHrN+748R3CHxJKpIbVn/j9Vn7g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:51:23.232 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:51:23.262 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:51:23.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:51:23.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:51:23.343 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:23.343 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:51:23.396 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:51:23.396 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 10:51:16.392741304 +0000 2024-03-29T10:51:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 10:48:36.390332818 +0000 2024-03-29T10:51:23.452 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 10:48:36.390332818 +0000 2024-03-29T10:51:23.452 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 10:48:36.390332818 +0000 2024-03-29T10:51:23.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:51:23.517 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T10:51:23.517 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T10:51:23.518 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.7975e-05 s, 5.2 MB/s 2024-03-29T10:51:23.519 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:51:23.577 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 10:51:16.393741289 +0000 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 10:48:36.716327614 +0000 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 10:48:36.716327614 +0000 2024-03-29T10:51:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 10:48:36.716327614 +0000 2024-03-29T10:51:23.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:51:23.714 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T10:51:23.715 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T10:51:23.715 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.2897e-05 s, 5.5 MB/s 2024-03-29T10:51:23.715 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:51:23.773 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-03-29T10:51:23.831 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:51:23.831 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:23.831 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:51:23.831 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:23.831 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:23.831 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 10:51:16.393741289 +0000 2024-03-29T10:51:23.832 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 10:48:37.049322299 +0000 2024-03-29T10:51:23.832 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 10:48:37.049322299 +0000 2024-03-29T10:51:23.832 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 10:48:37.049322299 +0000 2024-03-29T10:51:23.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:51:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T10:51:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T10:51:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 8.7431e-05 s, 5.9 MB/s 2024-03-29T10:51:23.902 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:51:23.968 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-03-29T10:51:24.037 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 10:51:16.393741289 +0000 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 10:48:37.381317000 +0000 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 10:48:37.381317000 +0000 2024-03-29T10:51:24.038 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 10:48:37.381317000 +0000 2024-03-29T10:51:24.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:51:24.115 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T10:51:24.115 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T10:51:24.116 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.4927e-05 s, 5.4 MB/s 2024-03-29T10:51:24.116 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:51:24.179 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:24.179 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:24.179 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:51:24.199 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:51:24.199 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-29T10:51:24.264 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-29 10:51:18.366859309 +0000 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-29 10:48:32.802792400 +0000 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-29 10:48:32.802792400 +0000 2024-03-29T10:51:24.265 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-29 10:48:32.802792400 +0000 2024-03-29T10:51:24.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:51:24.332 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-29T10:51:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-29T10:51:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000215441 s, 2.4 MB/s 2024-03-29T10:51:24.334 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:51:24.391 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-29T10:51:24.447 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:51:24.448 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:24.448 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:51:24.448 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:24.448 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:24.448 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-29 10:51:18.366859309 +0000 2024-03-29T10:51:24.448 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-29 10:48:33.145786424 +0000 2024-03-29T10:51:24.449 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-29 10:48:33.145786424 +0000 2024-03-29T10:51:24.449 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-29 10:48:33.145786424 +0000 2024-03-29T10:51:24.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:51:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-29T10:51:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-29T10:51:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000179007 s, 2.9 MB/s 2024-03-29T10:51:24.515 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:51:24.573 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-29T10:51:24.628 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-29 10:51:18.367859292 +0000 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-29 10:48:33.489780430 +0000 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-29 10:48:33.489780430 +0000 2024-03-29T10:51:24.629 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-29 10:48:33.489780430 +0000 2024-03-29T10:51:24.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:51:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-29T10:51:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-29T10:51:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000177627 s, 2.9 MB/s 2024-03-29T10:51:24.697 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:51:24.753 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-29T10:51:24.808 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-29 10:51:18.367859292 +0000 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-29 10:48:33.827774540 +0000 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-29 10:48:33.827774540 +0000 2024-03-29T10:51:24.809 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-29 10:48:33.827774540 +0000 2024-03-29T10:51:24.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:51:24.878 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-29T10:51:24.878 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-29T10:51:24.878 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000270386 s, 1.9 MB/s 2024-03-29T10:51:24.879 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:51:24.938 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:24.938 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:51:24.938 INFO:tasks.ceph:Generating config... 2024-03-29T10:51:24.939 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:51:24.939 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:51:24.939 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:51:24.939 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:51:24.940 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:51:24.940 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:51:24.940 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:51:24.940 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:51:24.940 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:51:24.940 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:51:24.940 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:51:24.941 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:51:24.942 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:51:24.942 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:51:24.942 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:51:24.942 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:51:24.942 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:51:24.943 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:51:24.944 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:51:24.945 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:51:24.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:24.999 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:51:25.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:51:25.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:51:25.078 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.45:6789]'), ('mon.c', '[v1:172.21.15.45:6790]'), ('mon.b', '[v1:172.21.15.116:6789]')] 2024-03-29T10:51:25.082 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.116:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:51:25.084 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:25.084 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:51:25.136 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.45:6789]' --addv c '[v1:172.21.15.45:6790]' --addv b '[v1:172.21.15.116:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:25.224 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:25.224 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: generated fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:51:25.224 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-03-29T10:51:25.224 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 0 2024-03-29T10:51:25.224 INFO:teuthology.orchestra.run.smithi045.stdout:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:min_mon_release 17 (quincy) 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:election_strategy: 1 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:51:25.225 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:51:25.226 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:51:25.241 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017... 2024-03-29T10:51:25.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:51:25.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:51:25.342 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-03-29T10:51:25.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-03-29T10:51:25.344 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-03-29T10:51:25.345 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.116:6789]" 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: lockdep = True 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: mon client directed command retry = 5 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: mon election default strategy = 1 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: ms bind msgr2 = False 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject socket failures = 5000 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: ms type = async 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:51:25.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-03-29T10:51:25.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression algorithm = zstd 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression mode = aggressive 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore zero block detection = True 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 20 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 20 2024-03-29T10:51:25.348 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 10 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd class default list = * 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd class load list = * 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd client watch timeout = 120 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify cached snaps = True 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify missing on start = True 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock override recovery settings = True 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-03-29T10:51:25.349 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue = debug_random 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue cut off = debug_random 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: osd shutdown pgref assert = True 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T10:51:25.350 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.351 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon min osdmap epochs = 50 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune interval = 2 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune min = 15 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub interval = 300 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on pool no app = False 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: paxos service trim min = 10 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:51:25.352 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: debug objecter = 20 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout: debug rados = 20 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-03-29T10:51:25.353 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-03-29T10:51:25.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:51:25.355 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-03-29T10:51:25.356 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.116:6789]" 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: lockdep = True 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: mon client directed command retry = 5 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: mon election default strategy = 1 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: ms bind msgr2 = False 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject socket failures = 5000 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: ms type = async 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.357 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-03-29T10:51:25.358 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression algorithm = zstd 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression mode = aggressive 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-03-29T10:51:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore zero block detection = True 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 20 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 20 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: debug monc = 20 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 10 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:51:25.360 INFO:teuthology.orchestra.run.smithi116.stdout: osd class default list = * 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd class load list = * 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd client watch timeout = 120 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify cached snaps = True 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify missing on start = True 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock override recovery settings = True 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd op queue = debug_random 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd op queue cut off = debug_random 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: osd shutdown pgref assert = True 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-03-29T10:51:25.361 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-03-29T10:51:25.362 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:51:25.363 INFO:teuthology.orchestra.run.smithi116.stdout: mon min osdmap epochs = 50 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune interval = 2 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune min = 15 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: mon scrub interval = 300 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on pool no app = False 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: paxos service trim min = 10 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: debug monc = 20 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-03-29T10:51:25.364 INFO:teuthology.orchestra.run.smithi116.stdout: debug objecter = 20 2024-03-29T10:51:25.365 INFO:teuthology.orchestra.run.smithi116.stdout: debug rados = 20 2024-03-29T10:51:25.365 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-03-29T10:51:25.365 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-03-29T10:51:25.365 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-03-29T10:51:25.366 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:51:25.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:51:25.464 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:51:25.464 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:25.464 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:51:25.479 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:25.479 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:51:25.534 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T10:51:25.535 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:25.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:51:25.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:51:25.617 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:25.617 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:25.672 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T10:51:25.672 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:25.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:51:25.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:51:25.714 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:25.714 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:25.770 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:51:25.770 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:51:25.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:51:25.825 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:51:25.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:51:25.889 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:51:25.891 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:51:25.892 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:51:25.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:51:25.934 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:51:25.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:51:25.991 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:51:26.007 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:51:26.007 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:51:26.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:26.034 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:26.035 INFO:tasks.ceph:role: osd.0 2024-03-29T10:51:26.035 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T10:51:26.035 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:51:26.101 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:26.101 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:26.101 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:26.101 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:26.101 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:26.102 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:26.102 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:26.102 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:26.102 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:26.102 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:26.134 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T10:51:26.136 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T10:51:26.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:26.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:26.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:26.350 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:26.350 INFO:tasks.ceph:role: osd.1 2024-03-29T10:51:26.350 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T10:51:26.350 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:51:26.416 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:26.416 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:26.416 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:26.416 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:26.417 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:26.417 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:26.417 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:26.417 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:26.417 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:26.417 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:26.448 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T10:51:26.450 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T10:51:26.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:26.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:26.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:26.659 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:26.659 INFO:tasks.ceph:role: osd.2 2024-03-29T10:51:26.659 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T10:51:26.660 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:51:26.725 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:26.726 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:26.758 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T10:51:26.760 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T10:51:26.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:26.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:26.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:26.975 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:26.975 INFO:tasks.ceph:role: osd.3 2024-03-29T10:51:26.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T10:51:26.975 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:51:27.042 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:27.042 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:27.042 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:27.042 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:27.043 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:27.043 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:27.043 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:27.043 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:27.043 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:27.043 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:27.074 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T10:51:27.076 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T10:51:27.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:27.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:27.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:27.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:27.335+0000 7f21b1812640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:51:27.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:27.335+0000 7f21b1812640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:51:27.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:27.335+0000 7f21b1812640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:51:27.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:27.335+0000 7f21b1812640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:51:27.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:27.335+0000 7f21b1812640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:51:27.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:27.336+0000 7f21b1812640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:51:30.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:30.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:30.306+0000 7f51a66f7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:51:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:30.306+0000 7f51a66f7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:51:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:30.306+0000 7f51a66f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:51:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:30.306+0000 7f51a66f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:51:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:30.306+0000 7f51a66f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:51:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:30.307+0000 7f51a66f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:51:33.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:33.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:33.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:33.268+0000 7fec6ec25640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:51:33.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:33.268+0000 7fec6ec25640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:51:33.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:33.268+0000 7fec6ec25640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:51:33.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:33.268+0000 7fec6ec25640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:51:33.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:33.268+0000 7fec6ec25640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:51:33.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:33.269+0000 7fec6ec25640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:51:36.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:36.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:36.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:36.231+0000 7f32fc593640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:51:36.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:36.231+0000 7f32fc593640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:51:36.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:36.231+0000 7f32fc593640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:51:36.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:36.231+0000 7f32fc593640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:51:36.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:36.231+0000 7f32fc593640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:51:36.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:36.231+0000 7f32fc593640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:51:39.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:39.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:39.130 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:39.130 INFO:tasks.ceph:role: osd.4 2024-03-29T10:51:39.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T10:51:39.130 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:51:39.208 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:39.208 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:39.209 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:39.724 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-03-29T10:51:39.726 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-03-29T10:51:39.727 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:39.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:39.914 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:39.984 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:39.984 INFO:tasks.ceph:role: osd.5 2024-03-29T10:51:39.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T10:51:39.984 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:51:40.059 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:40.060 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:40.504 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-03-29T10:51:40.506 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-03-29T10:51:40.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:40.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:40.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:40.695 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:40.696 INFO:tasks.ceph:role: osd.6 2024-03-29T10:51:40.696 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T10:51:40.696 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:51:40.770 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:40.770 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:40.771 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:41.264 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-03-29T10:51:41.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-03-29T10:51:41.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:41.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:41.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:41.513 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:41.513 INFO:tasks.ceph:role: osd.7 2024-03-29T10:51:41.514 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T10:51:41.514 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:51:41.587 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:41.587 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:41.587 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:41.588 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:42.085 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-03-29T10:51:42.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-03-29T10:51:42.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:42.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:42.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:42.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:42.375+0000 7fc0a761f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:51:42.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:42.375+0000 7fc0a761f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:51:42.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:42.375+0000 7fc0a761f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:51:42.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:42.375+0000 7fc0a761f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:51:42.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:42.375+0000 7fc0a761f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:51:42.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:42.376+0000 7fc0a761f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:51:45.207 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:45.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:45.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:45.346+0000 7f76bdcc8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:51:45.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:45.346+0000 7f76bdcc8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:51:45.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:45.346+0000 7f76bdcc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:51:45.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:45.346+0000 7f76bdcc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:51:45.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:45.346+0000 7f76bdcc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:51:45.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:45.347+0000 7f76bdcc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:51:48.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:48.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:48.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:48.309+0000 7f0aee415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:51:48.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:48.310+0000 7f0aee415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:51:48.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:48.310+0000 7f0aee415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:51:48.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:48.310+0000 7f0aee415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:51:48.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:48.310+0000 7f0aee415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:51:48.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:48.310+0000 7f0aee415640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:51:51.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:51.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:51.279+0000 7fbce0a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:51:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:51.279+0000 7fbce0a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:51:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:51.279+0000 7fbce0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:51:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:51.279+0000 7fbce0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:51:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:51.279+0000 7fbce0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:51:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:51:51.280+0000 7fbce0a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:51:54.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:54.154 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:51:54.154 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:54.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:51:54.187 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:54.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:51:54.258 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:54.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:51:54.329 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:54.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:51:54.399 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:54.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:51:54.473 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:54.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:51:54.504 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:54.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:51:54.575 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:54.575 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:51:54.645 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:54.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:51:54.715 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:54.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:51:54.785 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:51:54.785 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:51:54.799 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:51:54.799 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:51:54.841 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:51:54.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:51:54.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:51:54.884 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.y] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAtnQZmJmVWMhAAFA0LgY0VhHszw3pwtjgw4A== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAvnQZmF3EOFBAA/QR9RpvUaSCydjIbowFo/Q== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAynQZm0RlUEhAAUuPz3Adg5mdd13/Nud3JhA== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQA1nQZmb2UOEBAAg1kvpWugqmlRnSJOyhKV1A== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQA4nQZmxZ3XDRAAzH8XEoCR5XorBh8fDt3uiQ== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAtnQZmNMQqNhAALCMcEe4KNDYRalp9fcqWJA== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQA+nQZmyxFvFhAA1NS4G4fyGJ9+SVnyGB6wEg== 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-03-29T10:51:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBBnQZmX6exFBAAT4deWGjvhmcvqxtJrTNERQ== 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.6] 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBEnQZmuSOGEhAAZXWxsIaj87aJQuhmHUrcxw== 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.7] 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBHnQZmycW0EBAAkoBrL0zFXeKjJEnF10zb+w== 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAtnQZmkfncOBAAr65K8CWMTZUQmm7KhO3FFg== 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-03-29T10:51:54.886 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAunQZm+aCpABAA4sozbDhskS3w/dRA4rj31Q== 2024-03-29T10:51:54.908 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2024-03-29T10:51:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAtnQZmJmVWMhAAFA0LgY0VhHszw3pwtjgw4A== 2024-03-29T10:51:54.908 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAvnQZmF3EOFBAA/QR9RpvUaSCydjIbowFo/Q== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAynQZm0RlUEhAAUuPz3Adg5mdd13/Nud3JhA== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA1nQZmb2UOEBAAg1kvpWugqmlRnSJOyhKV1A== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA4nQZmxZ3XDRAAzH8XEoCR5XorBh8fDt3uiQ== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAtnQZmNMQqNhAALCMcEe4KNDYRalp9fcqWJA== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA+nQZmyxFvFhAA1NS4G4fyGJ9+SVnyGB6wEg== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBBnQZmX6exFBAAT4deWGjvhmcvqxtJrTNERQ== 2024-03-29T10:51:54.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBEnQZmuSOGEhAAZXWxsIaj87aJQuhmHUrcxw== 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBHnQZmycW0EBAAkoBrL0zFXeKjJEnF10zb+w== 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAtnQZmkfncOBAAr65K8CWMTZUQmm7KhO3FFg== 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2024-03-29T10:51:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAunQZm+aCpABAA4sozbDhskS3w/dRA4rj31Q== 2024-03-29T10:51:54.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:54.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:55.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:55.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:55.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:55.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:55.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:55.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:55.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:55.607 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:51:55.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:51:55.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:55.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:51:55.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:51:55.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:55.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:51:55.947 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:51:55.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:56.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:51:56.098 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:56.101 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:56.153 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:51:56.153 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:51:56.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:51:56.156 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:51:56.156 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:51:56.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:51:56.158 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:51:56.158 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:51:56.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:51:56.195 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:51:56.196 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:51:56.196 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:51:56.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:51:56.198 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:51:56.198 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:51:56.198 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:51:56.200 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:51:56.200 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:51:56.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.320+0000 7f359e20f640 10 monclient: get_monmap_and_config 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.320+0000 7f359e20f640 10 monclient: build_initial_monmap 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.320+0000 7f359e20f640 10 monclient: monmap: 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:56.321561+0000 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:56.321561+0000 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:56.302 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:51:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:51:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:51:56.303 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:51:56.303 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:51:56.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.321+0000 7f359e20f640 1 Processor -- start 2024-03-29T10:51:56.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.321+0000 7f359e20f640 1 -- start start 2024-03-29T10:51:56.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.321+0000 7f359e20f640 10 monclient: init 2024-03-29T10:51:56.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:56.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:56.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient(hunting): picked mon.noname-a con 0x7f35980b8240 addr v1:172.21.15.45:6789/0 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient(hunting): picked mon.noname-b con 0x7f35980b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient(hunting): picked mon.noname-c con 0x7f359809c8c0 addr v1:172.21.15.116:6789/0 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359809bd80 con 0x7f35980b8240 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359809b960 con 0x7f359809c8c0 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35980b2ec0 con 0x7f35980b1f60 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f359e20f640 10 monclient(hunting): authenticate will time out at 790.591170s 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f35977fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f35980b8240 0x7f359813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:32958/0 (socket says 172.21.15.45:32958) 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f3596ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f35980b1f60 0x7f359809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53650/0 (socket says 172.21.15.45:53650) 2024-03-29T10:51:56.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:56.322+0000 7f35977fe640 1 -- 172.21.15.45:0/3817945705 learned_addr learned my addr 172.21.15.45:0/3817945705 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:51:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): tick 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 monclient(hunting): continuing hunt 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 -- 172.21.15.45:0/3817945705 >> v1:172.21.15.45:6790/0 conn(0x7f35980b1f60 legacy=0x7f359809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 -- 172.21.15.45:0/3817945705 >> v1:172.21.15.116:6789/0 conn(0x7f359809c8c0 legacy=0x7f359814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 -- 172.21.15.45:0/3817945705 >> v1:172.21.15.45:6789/0 conn(0x7f35980b8240 legacy=0x7f359813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): picked mon.noname-a con 0x7f35840036e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): picked mon.noname-c con 0x7f35840072d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): picked mon.noname-b con 0x7f358400b000 addr v1:172.21.15.45:6790/0 2024-03-29T10:51:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359813fb60 con 0x7f35840036e0 2024-03-29T10:51:59.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358400fe10 con 0x7f35840072d0 2024-03-29T10:51:59.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3584011050 con 0x7f358400b000 2024-03-29T10:51:59.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:51:59.322+0000 7f35957fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031361552 0 0) 0x7f359813fb60 con 0x7f35840036e0 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580003520 con 0x7f35840036e0 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564177569 0 0) 0x7f3584011050 con 0x7f358400b000 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:52:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f359813fb60 con 0x7f358400b000 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491703227 0 0) 0x7f358400fe10 con 0x7f35840072d0 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.489+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584011050 con 0x7f35840072d0 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 797528645 0 0) 0x7f359813fb60 con 0x7f358400b000 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 >> v1:172.21.15.116:6789/0 conn(0x7f35840072d0 legacy=0x7f3584009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 >> v1:172.21.15.45:6789/0 conn(0x7f35840036e0 legacy=0x7f3584005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 monclient: found mon.noname-b 2024-03-29T10:52:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584012290 con 0x7f358400b000 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3590001a10 con 0x7f358400b000 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: dump: 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f359e20f640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f359e20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3590001b70 con 0x7f358400b000 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 1 -- 172.21.15.45:0/3817945705 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3590001f80 con 0x7f358400b000 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f359e20f640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f359e20f640 4 set_mon_vals no callback set 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.490+0000 7f35967fc640 10 monclient: dump: 2024-03-29T10:52:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 10 monclient: shutdown 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 1 -- 172.21.15.45:0/3817945705 >> v1:172.21.15.45:6790/0 conn(0x7f358400b000 legacy=0x7f358400d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 1 -- 172.21.15.45:0/3817945705 shutdown_connections 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 1 -- 172.21.15.45:0/3817945705 >> 172.21.15.45:0/3817945705 conn(0x7f3598066b00 msgr2=0x7f3598066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 1 -- 172.21.15.45:0/3817945705 shutdown_connections 2024-03-29T10:52:01.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.491+0000 7f359e20f640 1 -- 172.21.15.45:0/3817945705 wait complete. 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.492+0000 7f359e20f640 10 monclient: build_initial_monmap 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.492+0000 7f359e20f640 10 monclient: monmap: 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:01.485 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 1 librados: starting msgr at 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 1 librados: starting objecter 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 1 Processor -- start 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 1 -- start start 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 1 librados: setting wanted keys 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 1 librados: calling monclient init 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.493+0000 7f359e20f640 10 monclient: init 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f359e20f640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f359e20f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f359e20f640 10 monclient(hunting): picked mon.noname-b con 0x7f35840036e0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f359e20f640 10 monclient(hunting): picked mon.noname-c con 0x7f35840072d0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f359e20f640 10 monclient(hunting): picked mon.noname-a con 0x7f358400b000 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:01.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f35977fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f35840036e0 0x7f35981d7f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:43178/0 (socket says 172.21.15.45:43178) 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f35977fe640 1 -- 172.21.15.45:0/4233106400 learned_addr learned my addr 172.21.15.45:0/4233106400 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35981c9000 con 0x7f358400b000 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f357ffff640 10 client.?.objecter ms_handle_connect 0x7f35840036e0 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f357ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f357ffff640 10 client.?.objecter ms_handle_connect 0x7f35840072d0 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.494+0000 7f357ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35981c9270 con 0x7f35840036e0 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 10 client.?.objecter ms_handle_connect 0x7f358400b000 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35981d8680 con 0x7f35840072d0 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f359e20f640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f359e20f640 10 monclient(hunting): authenticate will time out at 795.764225s 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694246262 0 0) 0x7f35981d8680 con 0x7f35840072d0 2024-03-29T10:52:01.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.495+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358400e500 con 0x7f35840072d0 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2139627917 0 0) 0x7f358400e500 con 0x7f35840072d0 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35981d8680 con 0x7f35840072d0 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f358c0017c0 con 0x7f35840072d0 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient(hunting): dump: 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1453662579 0 0) 0x7f35981d8680 con 0x7f35840072d0 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 >> v1:172.21.15.116:6789/0 conn(0x7f35840036e0 legacy=0x7f35981d7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 >> v1:172.21.15.45:6789/0 conn(0x7f358400b000 legacy=0x7f35981e3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 monclient: found mon.c 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35981d8810 con 0x7f35840072d0 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: _finish_auth 0 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 5 monclient: authenticate success, global_id 4109 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 10 monclient: _renew_subs 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35980adfe0 con 0x7f35840072d0 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:01.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 10 monclient: _renew_subs 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35980ae290 con 0x7f35840072d0 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f359e20f640 1 librados: init done 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f358c0047e0 con 0x7f35840072d0 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f358c004af0 con 0x7f35840072d0 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f3595ffb640 4 set_mon_vals no callback set 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 monclient: dump: 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:01.491 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f358c005020 con 0x7f35840072d0 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2484456995 0 0) 0x7f358c0065e0 con 0x7f35840072d0 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 10 client.4109.objecter ms_dispatch 0x7f3598000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 3 client.4109.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 3 client.4109.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.496+0000 7f357ffff640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.497+0000 7f359e20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.497+0000 7f359e20f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.497+0000 7f359e20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:01.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.497+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3578005080 con 0x7f35840072d0 2024-03-29T10:52:01.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.500+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f358c0067f0 con 0x7f35840072d0 2024-03-29T10:52:01.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.500+0000 7f357ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:01.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.500+0000 7f357ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.572+0000 7f359e20f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:52:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.572+0000 7f359e20f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:52:01.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.572+0000 7f359e20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:01.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.572+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3578005330 con 0x7f35840072d0 2024-03-29T10:52:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.575+0000 7f357ffff640 1 -- 172.21.15.45:0/4233106400 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f358c005630 con 0x7f35840072d0 2024-03-29T10:52:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.575+0000 7f357ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:52:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.575+0000 7f357ffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:52:01.556 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 10 librados: watch_flush enter 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 10 librados: watch_flush exit 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 10 client.4109.objecter successfully canceled tick 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 10 monclient: shutdown 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 >> v1:172.21.15.45:6790/0 conn(0x7f35840072d0 legacy=0x7f3598139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 shutdown_connections 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 >> 172.21.15.45:0/4233106400 conn(0x7f3598066b00 msgr2=0x7f35980b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 shutdown_connections 2024-03-29T10:52:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 1 -- 172.21.15.45:0/4233106400 wait complete. 2024-03-29T10:52:01.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:01.576+0000 7f359e20f640 1 librados: shutdown 2024-03-29T10:52:01.568 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:52:01.569 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:52:01.569 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:52:01.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:52:01.596 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:52:01.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:52:01.664 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:52:01.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:52:01.731 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T10:52:01.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:52:01.798 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:52:01.798 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:52:01.826 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:52:01.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:52:01.895 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:52:01.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:52:01.962 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-29T10:52:01.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:52:01.968 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:01.987+0000 7fc21ca3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:52:01.972 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:01.991+0000 7ffbd0636040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:52:01.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 3383bc44-6790-4a17-9217-d8bfe1f9974a 0 2024-03-29T10:52:02.024 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.043+0000 7fc21ca3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:52:02.030 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.049+0000 7ffbd0636040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:52:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.146+0000 7f3e50efb640 10 monclient: get_monmap_and_config 2024-03-29T10:52:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.146+0000 7f3e50efb640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.147+0000 7f3e50efb640 10 monclient: monmap: 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:02.147884+0000 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:02.147884+0000 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.147+0000 7f3e50efb640 1 Processor -- start 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.147+0000 7f3e50efb640 1 -- start start 2024-03-29T10:52:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.147+0000 7f3e50efb640 10 monclient: init 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4c09ac40 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4c09c870 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4c0b4360 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c09dec0 con 0x7f3e4c09c870 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c09da30 con 0x7f3e4c09ac40 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0b39c0 con 0x7f3e4c0b4360 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 10 monclient(hunting): authenticate will time out at 804.454507s 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3e4c09ac40 0x7f3e4c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43460/0 (socket says 172.21.15.116:43460) 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4a59c640 1 -- 172.21.15.116:0/2819898929 learned_addr learned my addr 172.21.15.116:0/2819898929 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274604392 0 0) 0x7f3e4c09da30 con 0x7f3e4c09ac40 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e34003520 con 0x7f3e4c09ac40 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741703582 0 0) 0x7f3e40001110 con 0x7f3e4c09c870 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c09da30 con 0x7f3e4c09c870 2024-03-29T10:52:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119462794 0 0) 0x7f3e4c0b39c0 con 0x7f3e4c0b4360 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e40001110 con 0x7f3e4c0b4360 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1710240977 0 0) 0x7f3e34003520 con 0x7f3e4c09ac40 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 >> v1:172.21.15.45:6790/0 conn(0x7f3e4c0b4360 legacy=0x7f3e4c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 >> v1:172.21.15.45:6789/0 conn(0x7f3e4c09c870 legacy=0x7f3e4c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 monclient: found mon.noname-c 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c0b5160 con 0x7f3e4c09ac40 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e3c001690 con 0x7f3e4c09ac40 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e50efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.148+0000 7f3e4959a640 10 monclient: dump: 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e3c001870 con 0x7f3e4c09ac40 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e4959a640 1 -- 172.21.15.116:0/2819898929 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e3c001c80 con 0x7f3e4c09ac40 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 4 set_mon_vals no callback set 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e4959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e4959a640 10 monclient: dump: 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 10 monclient: shutdown 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 1 -- 172.21.15.116:0/2819898929 >> v1:172.21.15.116:6789/0 conn(0x7f3e4c09ac40 legacy=0x7f3e4c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 1 -- 172.21.15.116:0/2819898929 shutdown_connections 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 1 -- 172.21.15.116:0/2819898929 >> 172.21.15.116:0/2819898929 conn(0x7f3e4c066b00 msgr2=0x7f3e4c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 1 -- 172.21.15.116:0/2819898929 shutdown_connections 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.149+0000 7f3e50efb640 1 -- 172.21.15.116:0/2819898929 wait complete. 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 10 monclient: monmap: 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 1 librados: starting msgr at 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 1 librados: starting objecter 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 1 Processor -- start 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 1 -- start start 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 1 librados: setting wanted keys 2024-03-29T10:52:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 1 librados: calling monclient init 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.150+0000 7f3e50efb640 10 monclient: init 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4c09ac40 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4c1ccb70 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4c1425f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c1cd520 con 0x7f3e4c1ccb70 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c0acd00 con 0x7f3e4c09ac40 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c1cd000 con 0x7f3e4c1425f0 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e50efb640 10 monclient(hunting): authenticate will time out at 804.457478s 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e4a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3e4c09ac40 0x7f3e4c1da690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43476/0 (socket says 172.21.15.116:43476) 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e4a59c640 1 -- 172.21.15.116:0/1631296332 learned_addr learned my addr 172.21.15.116:0/1631296332 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 client.?.objecter ms_handle_connect 0x7f3e4c09ac40 2024-03-29T10:52:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 client.?.objecter ms_handle_connect 0x7f3e4c1425f0 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 client.?.objecter ms_handle_connect 0x7f3e4c1ccb70 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69274345 0 0) 0x7f3e4c0acd00 con 0x7f3e4c09ac40 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c004eb0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277261142 0 0) 0x7f3e4c1cd000 con 0x7f3e4c1425f0 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c0acd00 con 0x7f3e4c1425f0 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518976315 0 0) 0x7f3e4c1cd520 con 0x7f3e4c1ccb70 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c1cd000 con 0x7f3e4c1ccb70 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113293437 0 0) 0x7f3e2c004eb0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e4c1cd520 con 0x7f3e4c09ac40 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e3c001bc0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260926095 0 0) 0x7f3e4c0acd00 con 0x7f3e4c1425f0 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e2c004eb0 con 0x7f3e4c1425f0 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e44001a70 con 0x7f3e4c1425f0 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.151+0000 7f3e3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462034453 0 0) 0x7f3e4c1cd000 con 0x7f3e4c1ccb70 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e4c0acd00 con 0x7f3e4c1ccb70 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3811818496 0 0) 0x7f3e4c1cd520 con 0x7f3e4c09ac40 2024-03-29T10:52:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 >> v1:172.21.15.45:6790/0 conn(0x7f3e4c1425f0 legacy=0x7f3e4c1e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 >> v1:172.21.15.45:6789/0 conn(0x7f3e4c1ccb70 legacy=0x7f3e4c13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 monclient: found mon.b 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c0b1af0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 monclient: _renew_subs 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c1cc300 con 0x7f3e4c09ac40 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 monclient: _renew_subs 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c1daf80 con 0x7f3e4c09ac40 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 1 librados: init done 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e3c004bf0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e3c004f00 con 0x7f3e4c09ac40 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 monclient: dump: 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e48d99640 4 set_mon_vals no callback set 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3e3c0053f0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 4122053045 0 0) 0x7f3e3c006870 con 0x7f3e4c09ac40 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 10 client.4111.objecter ms_dispatch 0x7f3e4c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e3affd640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.152+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e30005080 con 0x7f3e4c09ac40 2024-03-29T10:52:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.155+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3e3c006a80 con 0x7f3e4c09ac40 2024-03-29T10:52:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.155+0000 7f3e3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.155+0000 7f3e3affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:02.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.227+0000 7f3e50efb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3383bc44-6790-4a17-9217-d8bfe1f9974a", "id": 0}] 2024-03-29T10:52:02.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.227+0000 7f3e50efb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3383bc44-6790-4a17-9217-d8bfe1f9974a", "id": 0}] 2024-03-29T10:52:02.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.227+0000 7f3e50efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.227+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3383bc44-6790-4a17-9217-d8bfe1f9974a", "id": 0} v 0) v1 -- 0x7f3e300058b0 con 0x7f3e4c09ac40 2024-03-29T10:52:02.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.232+0000 7f3e3affd640 1 -- 172.21.15.116:0/1631296332 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3383bc44-6790-4a17-9217-d8bfe1f9974a", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2310024919 0 732707515) 0x7f3e3c005910 con 0x7f3e4c09ac40 2024-03-29T10:52:02.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.232+0000 7f3e3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3383bc44-6790-4a17-9217-d8bfe1f9974a", "id": 0}] 2024-03-29T10:52:02.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.232+0000 7f3e3affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:02.214 INFO:teuthology.orchestra.run.smithi116.stdout:0 2024-03-29T10:52:02.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 10 librados: watch_flush enter 2024-03-29T10:52:02.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 10 librados: watch_flush exit 2024-03-29T10:52:02.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:52:02.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 10 monclient: shutdown 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 >> v1:172.21.15.116:6789/0 conn(0x7f3e4c09ac40 legacy=0x7f3e4c1da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 shutdown_connections 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 >> 172.21.15.116:0/1631296332 conn(0x7f3e4c066b00 msgr2=0x7f3e4c066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 shutdown_connections 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 1 -- 172.21.15.116:0/1631296332 wait complete. 2024-03-29T10:52:02.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.233+0000 7f3e50efb640 1 librados: shutdown 2024-03-29T10:52:02.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 105e5d19-e910-4a79-92e6-56d6a0706900 1 2024-03-29T10:52:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.343+0000 7fa298e86640 10 monclient: get_monmap_and_config 2024-03-29T10:52:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.343+0000 7fa298e86640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.343+0000 7fa298e86640 10 monclient: monmap: 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:02.344029+0000 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:02.344029+0000 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:02.325 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.343+0000 7fa298e86640 1 Processor -- start 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.343+0000 7fa298e86640 1 -- start start 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.343+0000 7fa298e86640 10 monclient: init 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient(hunting): picked mon.noname-b con 0x7fa29409ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient(hunting): picked mon.noname-c con 0x7fa29409c870 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient(hunting): picked mon.noname-a con 0x7fa2940b4360 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29409dec0 con 0x7fa2940b4360 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29409da30 con 0x7fa29409c870 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940b39c0 con 0x7fa29409ac40 2024-03-29T10:52:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa298e86640 10 monclient(hunting): authenticate will time out at 804.650723s 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa291d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa29409c870 0x7fa29413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43484/0 (socket says 172.21.15.116:43484) 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa291d9b640 1 -- 172.21.15.116:0/584970856 learned_addr learned my addr 172.21.15.116:0/584970856 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920230268 0 0) 0x7fa29409da30 con 0x7fa29409c870 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa29159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa29159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa29159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa29159a640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.344+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c003520 con 0x7fa29409c870 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147727795 0 0) 0x7fa28c0013c0 con 0x7fa2940b4360 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:52:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29409da30 con 0x7fa2940b4360 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742833616 0 0) 0x7fa2940b39c0 con 0x7fa29409ac40 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa28c0013c0 con 0x7fa29409ac40 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2595662205 0 0) 0x7fa27c003520 con 0x7fa29409c870 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 >> v1:172.21.15.45:6790/0 conn(0x7fa29409ac40 legacy=0x7fa2940b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 >> v1:172.21.15.45:6789/0 conn(0x7fa2940b4360 legacy=0x7fa29414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 monclient: found mon.noname-c 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940b5160 con 0x7fa29409c870 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa284001690 con 0x7fa29409c870 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: dump: 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa284001870 con 0x7fa29409c870 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 1 -- 172.21.15.116:0/584970856 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa284001c80 con 0x7fa29409c870 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 4 set_mon_vals no callback set 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa29159a640 10 monclient: dump: 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 10 monclient: shutdown 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 1 -- 172.21.15.116:0/584970856 >> v1:172.21.15.116:6789/0 conn(0x7fa29409c870 legacy=0x7fa29413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 1 -- 172.21.15.116:0/584970856 shutdown_connections 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 1 -- 172.21.15.116:0/584970856 >> 172.21.15.116:0/584970856 conn(0x7fa294066b00 msgr2=0x7fa294066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 1 -- 172.21.15.116:0/584970856 shutdown_connections 2024-03-29T10:52:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.345+0000 7fa298e86640 1 -- 172.21.15.116:0/584970856 wait complete. 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.346+0000 7fa298e86640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.346+0000 7fa298e86640 10 monclient: monmap: 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.346+0000 7fa298e86640 1 librados: starting msgr at 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.346+0000 7fa298e86640 1 librados: starting objecter 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 Processor -- start 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 -- start start 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 librados: setting wanted keys 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 librados: calling monclient init 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient: init 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient(hunting): picked mon.noname-a con 0x7fa29409ac40 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient(hunting): picked mon.noname-b con 0x7fa2940af410 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient(hunting): picked mon.noname-c con 0x7fa2941dafd0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940b1590 con 0x7fa29409ac40 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2941da3c0 con 0x7fa2940af410 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940b1800 con 0x7fa2941dafd0 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa298e86640 10 monclient(hunting): authenticate will time out at 804.653802s 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa291d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa2940af410 0x7fa2941da8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43494/0 (socket says 172.21.15.116:43494) 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa291d9b640 1 -- 172.21.15.116:0/1168467074 learned_addr learned my addr 172.21.15.116:0/1168467074 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa29259c640 1 --1- 172.21.15.116:0/1168467074 >> v1:172.21.15.45:6789/0 conn(0x7fa29409ac40 0x7fa2940afef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.116:60108/0 (socket says 172.21.15.116:60108) 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 client.?.objecter ms_handle_connect 0x7fa2940af410 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 client.?.objecter ms_handle_connect 0x7fa29409ac40 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 client.?.objecter ms_handle_connect 0x7fa2941dafd0 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8878717 0 0) 0x7fa2941da3c0 con 0x7fa2940af410 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa274004eb0 con 0x7fa2940af410 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286059097 0 0) 0x7fa2940b1590 con 0x7fa29409ac40 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:52:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.347+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2941da3c0 con 0x7fa29409ac40 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363620295 0 0) 0x7fa2940b1800 con 0x7fa2941dafd0 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2940b1590 con 0x7fa2941dafd0 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687695235 0 0) 0x7fa274004eb0 con 0x7fa2940af410 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2940b1800 con 0x7fa2940af410 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa284001bc0 con 0x7fa2940af410 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.346 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86408813 0 0) 0x7fa2941da3c0 con 0x7fa29409ac40 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa274004eb0 con 0x7fa29409ac40 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa2880018f0 con 0x7fa29409ac40 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2690952613 0 0) 0x7fa2940b1590 con 0x7fa2941dafd0 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2941da3c0 con 0x7fa2941dafd0 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 882897347 0 0) 0x7fa2940b1800 con 0x7fa2940af410 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 >> v1:172.21.15.45:6790/0 conn(0x7fa2941dafd0 legacy=0x7fa294139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 >> v1:172.21.15.45:6789/0 conn(0x7fa29409ac40 legacy=0x7fa2940afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 monclient: found mon.b 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940acd00 con 0x7fa2940af410 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 10 monclient: _renew_subs 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa294139750 con 0x7fa2940af410 2024-03-29T10:52:02.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 10 monclient: _renew_subs 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2941e3770 con 0x7fa2940af410 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa284004bf0 con 0x7fa2940af410 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa298e86640 1 librados: init done 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa284004f00 con 0x7fa2940af410 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 monclient: dump: 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.350 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa290d99640 4 set_mon_vals no callback set 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa284005470 con 0x7fa2940af410 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3763480123 0 0) 0x7fa2840068c0 con 0x7fa2940af410 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 10 client.4117.objecter ms_dispatch 0x7fa294000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:52:02.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.348+0000 7fa282ffd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.349+0000 7fa298e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.349+0000 7fa298e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.349+0000 7fa298e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.349+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2941da550 con 0x7fa2940af410 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.351+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa2941da550 con 0x7fa2940af410 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.351+0000 7fa282ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.351+0000 7fa282ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:02.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.422+0000 7fa298e86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "105e5d19-e910-4a79-92e6-56d6a0706900", "id": 1}] 2024-03-29T10:52:02.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.422+0000 7fa298e86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "105e5d19-e910-4a79-92e6-56d6a0706900", "id": 1}] 2024-03-29T10:52:02.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.422+0000 7fa298e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.422+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "105e5d19-e910-4a79-92e6-56d6a0706900", "id": 1} v 0) v1 -- 0x7fa2940b0600 con 0x7fa2940af410 2024-03-29T10:52:02.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.429+0000 7fa282ffd640 1 -- 172.21.15.116:0/1168467074 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "105e5d19-e910-4a79-92e6-56d6a0706900", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2267400055 0 940483276) 0x7fa284006ad0 con 0x7fa2940af410 2024-03-29T10:52:02.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.429+0000 7fa282ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "105e5d19-e910-4a79-92e6-56d6a0706900", "id": 1}] 2024-03-29T10:52:02.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.429+0000 7fa282ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:02.411 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 10 librados: watch_flush enter 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 10 librados: watch_flush exit 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 10 monclient: shutdown 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 >> v1:172.21.15.116:6789/0 conn(0x7fa2940af410 legacy=0x7fa2941da8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 shutdown_connections 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 >> 172.21.15.116:0/1168467074 conn(0x7fa294066b00 msgr2=0x7fa29413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 shutdown_connections 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 1 -- 172.21.15.116:0/1168467074 wait complete. 2024-03-29T10:52:02.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.430+0000 7fa298e86640 1 librados: shutdown 2024-03-29T10:52:02.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 5fa13e47-ddf2-43fb-87c2-1e22289c6cf5 2 2024-03-29T10:52:02.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.540+0000 7fd1a7d88640 10 monclient: get_monmap_and_config 2024-03-29T10:52:02.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.540+0000 7fd1a7d88640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.540+0000 7fd1a7d88640 10 monclient: monmap: 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:02.541688+0000 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:02.541688+0000 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.541+0000 7fd1a7d88640 1 Processor -- start 2024-03-29T10:52:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.541+0000 7fd1a7d88640 1 -- start start 2024-03-29T10:52:02.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.541+0000 7fd1a7d88640 10 monclient: init 2024-03-29T10:52:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a009ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a009c870 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a00b4360 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a009dec0 con 0x7fd1a00b4360 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a009da30 con 0x7fd1a009c870 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00b39c0 con 0x7fd1a009ac40 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 10 monclient(hunting): authenticate will time out at 804.848334s 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a5323640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd1a009c870 0x7fd1a013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43502/0 (socket says 172.21.15.116:43502) 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a5323640 1 -- 172.21.15.116:0/3707961227 learned_addr learned my addr 172.21.15.116:0/3707961227 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854201188 0 0) 0x7fd1a009da30 con 0x7fd1a009c870 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd18c003520 con 0x7fd1a009c870 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204703911 0 0) 0x7fd1a00b39c0 con 0x7fd1a009ac40 2024-03-29T10:52:02.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a009da30 con 0x7fd1a009ac40 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782851657 0 0) 0x7fd198001490 con 0x7fd1a00b4360 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a00b39c0 con 0x7fd1a00b4360 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175705932 0 0) 0x7fd18c003520 con 0x7fd1a009c870 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 >> v1:172.21.15.45:6790/0 conn(0x7fd1a009ac40 legacy=0x7fd1a00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 >> v1:172.21.15.45:6789/0 conn(0x7fd1a00b4360 legacy=0x7fd1a014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 monclient: found mon.noname-c 2024-03-29T10:52:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00b5160 con 0x7fd1a009c870 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd19c001690 con 0x7fd1a009c870 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a4b22640 10 monclient: dump: 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.542+0000 7fd1a7d88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd19c001870 con 0x7fd1a009c870 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a4b22640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 4 set_mon_vals no callback set 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a4b22640 1 -- 172.21.15.116:0/3707961227 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd19c001c80 con 0x7fd1a009c870 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a4b22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a4b22640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a4b22640 10 monclient: dump: 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 10 monclient: shutdown 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 1 -- 172.21.15.116:0/3707961227 >> v1:172.21.15.116:6789/0 conn(0x7fd1a009c870 legacy=0x7fd1a013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 1 -- 172.21.15.116:0/3707961227 shutdown_connections 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 1 -- 172.21.15.116:0/3707961227 >> 172.21.15.116:0/3707961227 conn(0x7fd1a0066b00 msgr2=0x7fd1a0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 1 -- 172.21.15.116:0/3707961227 shutdown_connections 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.543+0000 7fd1a7d88640 1 -- 172.21.15.116:0/3707961227 wait complete. 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 10 monclient: monmap: 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 1 librados: starting msgr at 2024-03-29T10:52:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 1 librados: starting objecter 2024-03-29T10:52:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 1 Processor -- start 2024-03-29T10:52:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 1 -- start start 2024-03-29T10:52:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 1 librados: setting wanted keys 2024-03-29T10:52:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 1 librados: calling monclient init 2024-03-29T10:52:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.544+0000 7fd1a7d88640 10 monclient: init 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a009ac40 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a00b06c0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a01dafb0 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a01da410 con 0x7fd1a01dafb0 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00b0b70 con 0x7fd1a009ac40 2024-03-29T10:52:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a013ffa0 con 0x7fd1a00b06c0 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient(hunting): authenticate will time out at 804.851291s 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a5b24640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd1a009ac40 0x7fd1a00b0fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43512/0 (socket says 172.21.15.116:43512) 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a5b24640 1 -- 172.21.15.116:0/2169718657 learned_addr learned my addr 172.21.15.116:0/2169718657 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 client.?.objecter ms_handle_connect 0x7fd1a009ac40 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 client.?.objecter ms_handle_connect 0x7fd1a01dafb0 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562904106 0 0) 0x7fd1a00b0b70 con 0x7fd1a009ac40 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:52:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004eb0 con 0x7fd1a009ac40 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 client.?.objecter ms_handle_connect 0x7fd1a00b06c0 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529994114 0 0) 0x7fd1a01da410 con 0x7fd1a01dafb0 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a00b0b70 con 0x7fd1a01dafb0 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376817392 0 0) 0x7fd1a013ffa0 con 0x7fd1a00b06c0 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a01da410 con 0x7fd1a00b06c0 2024-03-29T10:52:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024598447 0 0) 0x7fd184004eb0 con 0x7fd1a009ac40 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1a013ffa0 con 0x7fd1a009ac40 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd194001a60 con 0x7fd1a009ac40 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): dump: 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.546 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768821592 0 0) 0x7fd1a00b0b70 con 0x7fd1a01dafb0 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd184004eb0 con 0x7fd1a01dafb0 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1823723335 0 0) 0x7fd1a01da410 con 0x7fd1a00b06c0 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1a00b0b70 con 0x7fd1a00b06c0 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd198001a70 con 0x7fd1a01dafb0 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): dump: 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd19c001b80 con 0x7fd1a00b06c0 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient(hunting): dump: 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409313020 0 0) 0x7fd1a013ffa0 con 0x7fd1a009ac40 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 >> v1:172.21.15.45:6790/0 conn(0x7fd1a00b06c0 legacy=0x7fd1a01da8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 >> v1:172.21.15.45:6789/0 conn(0x7fd1a01dafb0 legacy=0x7fd1a01e1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 monclient: found mon.b 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00add50 con 0x7fd1a009ac40 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient: _renew_subs 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a01e2fa0 con 0x7fd1a009ac40 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient: _renew_subs 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a01e3310 con 0x7fd1a009ac40 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.545+0000 7fd1a7d88640 1 librados: init done 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd194001ca0 con 0x7fd1a009ac40 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd194003a40 con 0x7fd1a009ac40 2024-03-29T10:52:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd193fff640 4 set_mon_vals no callback set 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 10 monclient: dump: 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd194003fb0 con 0x7fd1a009ac40 2024-03-29T10:52:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 345944097 0 0) 0x7fd194005be0 con 0x7fd1a009ac40 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 10 client.4123.objecter ms_dispatch 0x7fd1a0000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1927fc640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1a7d88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1a7d88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1a7d88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.546+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188005080 con 0x7fd1a009ac40 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.548+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd194005da0 con 0x7fd1a009ac40 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.548+0000 7fd1927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.548+0000 7fd1927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:02.566 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.584+0000 7fc21ca3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:52:02.574 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.591+0000 7ffbd0636040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:52:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.619+0000 7fd1a7d88640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5fa13e47-ddf2-43fb-87c2-1e22289c6cf5", "id": 2}] 2024-03-29T10:52:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.619+0000 7fd1a7d88640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5fa13e47-ddf2-43fb-87c2-1e22289c6cf5", "id": 2}] 2024-03-29T10:52:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.619+0000 7fd1a7d88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.619+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5fa13e47-ddf2-43fb-87c2-1e22289c6cf5", "id": 2} v 0) v1 -- 0x7fd1880058b0 con 0x7fd1a009ac40 2024-03-29T10:52:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.622+0000 7fd1927fc640 1 -- 172.21.15.116:0/2169718657 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5fa13e47-ddf2-43fb-87c2-1e22289c6cf5", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2887675256 0 216599125) 0x7fd194004330 con 0x7fd1a009ac40 2024-03-29T10:52:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.622+0000 7fd1927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5fa13e47-ddf2-43fb-87c2-1e22289c6cf5", "id": 2}] 2024-03-29T10:52:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.622+0000 7fd1927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:02.604 INFO:teuthology.orchestra.run.smithi116.stdout:2 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 10 librados: watch_flush enter 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 10 librados: watch_flush exit 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 10 monclient: shutdown 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 >> v1:172.21.15.116:6789/0 conn(0x7fd1a009ac40 legacy=0x7fd1a00b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 shutdown_connections 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 >> 172.21.15.116:0/2169718657 conn(0x7fd1a0066b00 msgr2=0x7fd1a013ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 shutdown_connections 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 1 -- 172.21.15.116:0/2169718657 wait complete. 2024-03-29T10:52:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.623+0000 7fd1a7d88640 1 librados: shutdown 2024-03-29T10:52:02.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new aacf3eb4-f626-4306-9973-92189256dbec 3 2024-03-29T10:52:02.622 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.641+0000 7fc21ca3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:52:02.628 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.647+0000 7ffbd0636040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:52:02.668 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.687+0000 7fc21ca3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:52:02.676 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.694+0000 7ffbd0636040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.733+0000 7f3a27460640 10 monclient: get_monmap_and_config 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.733+0000 7f3a27460640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.733+0000 7f3a27460640 10 monclient: monmap: 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:02.734412+0000 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:02.734412+0000 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.715 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.716 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:02.716 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:02.716 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 1 Processor -- start 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 1 -- start start 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient: init 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2009ac40 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2009c870 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient(hunting): picked mon.noname-a con 0x7f3a200b4360 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2009dec0 con 0x7f3a200b4360 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2009da30 con 0x7f3a2009ac40 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200b39c0 con 0x7f3a2009c870 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.734+0000 7f3a27460640 10 monclient(hunting): authenticate will time out at 805.041106s 2024-03-29T10:52:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a251fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3a2009ac40 0x7f3a200b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43526/0 (socket says 172.21.15.116:43526) 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a251fc640 1 -- 172.21.15.116:0/2849431085 learned_addr learned my addr 172.21.15.116:0/2849431085 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530909872 0 0) 0x7f3a2009da30 con 0x7f3a2009ac40 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c003520 con 0x7f3a2009ac40 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880740979 0 0) 0x7f3a200b39c0 con 0x7f3a2009c870 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2009da30 con 0x7f3a2009c870 2024-03-29T10:52:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548368073 0 0) 0x7f3a18001490 con 0x7f3a200b4360 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a200b39c0 con 0x7f3a200b4360 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252994354 0 0) 0x7f3a0c003520 con 0x7f3a2009ac40 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 >> v1:172.21.15.45:6790/0 conn(0x7f3a2009c870 legacy=0x7f3a2013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 >> v1:172.21.15.45:6789/0 conn(0x7f3a200b4360 legacy=0x7f3a2014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 monclient: found mon.noname-c 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a200b5160 con 0x7f3a2009ac40 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3a1c001690 con 0x7f3a2009ac40 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a27460640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: dump: 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a27460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a1c001870 con 0x7f3a2009ac40 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 1 -- 172.21.15.116:0/2849431085 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3a1c001c80 con 0x7f3a2009ac40 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a27460640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a27460640 4 set_mon_vals no callback set 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.735+0000 7f3a17fff640 10 monclient: dump: 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 10 monclient: shutdown 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 1 -- 172.21.15.116:0/2849431085 >> v1:172.21.15.116:6789/0 conn(0x7f3a2009ac40 legacy=0x7f3a200b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 1 -- 172.21.15.116:0/2849431085 shutdown_connections 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 1 -- 172.21.15.116:0/2849431085 >> 172.21.15.116:0/2849431085 conn(0x7f3a20066b00 msgr2=0x7f3a20066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 1 -- 172.21.15.116:0/2849431085 shutdown_connections 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.736+0000 7f3a27460640 1 -- 172.21.15.116:0/2849431085 wait complete. 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient: monmap: 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 1 librados: starting msgr at 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 1 librados: starting objecter 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 1 Processor -- start 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 1 -- start start 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 1 librados: setting wanted keys 2024-03-29T10:52:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 1 librados: calling monclient init 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient: init 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2009ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient(hunting): picked mon.noname-a con 0x7f3a200acf60 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.737+0000 7f3a27460640 10 monclient(hunting): picked mon.noname-b con 0x7f3a200b0800 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a201da7b0 con 0x7f3a200acf60 2024-03-29T10:52:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200b0c70 con 0x7f3a200b0800 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a201db350 con 0x7f3a2009ac40 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 10 monclient(hunting): authenticate will time out at 805.044196s 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a259fd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3a200b0800 0x7f3a201e1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43534/0 (socket says 172.21.15.116:43534) 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a251fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3a2009ac40 0x7f3a200af9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.116:51866/0 (socket says 172.21.15.116:51866) 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a259fd640 1 -- 172.21.15.116:0/1594097668 learned_addr learned my addr 172.21.15.116:0/1594097668 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 client.?.objecter ms_handle_connect 0x7f3a200b0800 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 client.?.objecter ms_handle_connect 0x7f3a2009ac40 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 client.?.objecter ms_handle_connect 0x7f3a200acf60 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169995231 0 0) 0x7f3a200b0c70 con 0x7f3a200b0800 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a04004eb0 con 0x7f3a200b0800 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271795054 0 0) 0x7f3a201db350 con 0x7f3a2009ac40 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a200b0c70 con 0x7f3a2009ac40 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572150996 0 0) 0x7f3a201da7b0 con 0x7f3a200acf60 2024-03-29T10:52:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a201db350 con 0x7f3a200acf60 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224054656 0 0) 0x7f3a201db350 con 0x7f3a200acf60 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a201da7b0 con 0x7f3a200acf60 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3a100017e0 con 0x7f3a200acf60 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685486419 0 0) 0x7f3a200b0c70 con 0x7f3a2009ac40 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a201db350 con 0x7f3a2009ac40 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3a1c001b80 con 0x7f3a2009ac40 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1013236212 0 0) 0x7f3a04004eb0 con 0x7f3a200b0800 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a200b0c70 con 0x7f3a200b0800 2024-03-29T10:52:02.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3a180019c0 con 0x7f3a200b0800 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4086355529 0 0) 0x7f3a201da7b0 con 0x7f3a200acf60 2024-03-29T10:52:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 >> v1:172.21.15.45:6790/0 conn(0x7f3a2009ac40 legacy=0x7f3a200af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 >> v1:172.21.15.116:6789/0 conn(0x7f3a200b0800 legacy=0x7f3a201e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 monclient: found mon.a 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a201db4e0 con 0x7f3a200acf60 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 5 monclient: authenticate success, global_id 4134 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 10 monclient: _renew_subs 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.738+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a201e3240 con 0x7f3a200acf60 2024-03-29T10:52:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a10001c90 con 0x7f3a200acf60 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3a10003610 con 0x7f3a200acf60 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 10 monclient: dump: 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.741 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 10 monclient: _renew_subs 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a177fe640 4 set_mon_vals no callback set 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a201e35b0 con 0x7f3a200acf60 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3a10003b00 con 0x7f3a200acf60 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 1 librados: init done 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1396886299 0 0) 0x7f3a10004630 con 0x7f3a200acf60 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 10 client.4134.objecter ms_dispatch 0x7f3a20000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a15ffb640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:02.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.739+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2009b020 con 0x7f3a200acf60 2024-03-29T10:52:02.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.742+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3a10003db0 con 0x7f3a200acf60 2024-03-29T10:52:02.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.742+0000 7f3a15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.742+0000 7f3a15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:02.744 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.752+0000 7ffbd0636040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:52:02.744 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.743+0000 7fc21ca3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:52:02.768 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.787+0000 7fc21ca3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:52:02.778 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.796+0000 7ffbd0636040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:52:02.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.813+0000 7f3a27460640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aacf3eb4-f626-4306-9973-92189256dbec", "id": 3}] 2024-03-29T10:52:02.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.813+0000 7f3a27460640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aacf3eb4-f626-4306-9973-92189256dbec", "id": 3}] 2024-03-29T10:52:02.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.813+0000 7f3a27460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:02.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.813+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aacf3eb4-f626-4306-9973-92189256dbec", "id": 3} v 0) v1 -- 0x7f3a2006ab60 con 0x7f3a200acf60 2024-03-29T10:52:02.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.815+0000 7f3a15ffb640 1 -- 172.21.15.116:0/1594097668 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aacf3eb4-f626-4306-9973-92189256dbec", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1368991214 0 525046306) 0x7f3a10004010 con 0x7f3a200acf60 2024-03-29T10:52:02.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.815+0000 7f3a15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aacf3eb4-f626-4306-9973-92189256dbec", "id": 3}] 2024-03-29T10:52:02.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.815+0000 7f3a15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:02.797 INFO:teuthology.orchestra.run.smithi116.stdout:3 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 10 librados: watch_flush enter 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 10 librados: watch_flush exit 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 10 client.4134.objecter successfully canceled tick 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 10 monclient: shutdown 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 >> v1:172.21.15.45:6789/0 conn(0x7f3a200acf60 legacy=0x7f3a201dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 shutdown_connections 2024-03-29T10:52:02.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 >> 172.21.15.116:0/1594097668 conn(0x7f3a20066b00 msgr2=0x7f3a200b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 shutdown_connections 2024-03-29T10:52:02.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 1 -- 172.21.15.116:0/1594097668 wait complete. 2024-03-29T10:52:02.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.816+0000 7f3a27460640 1 librados: shutdown 2024-03-29T10:52:02.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3 4 2024-03-29T10:52:02.889 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.907+0000 7fc21ca3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:52:02.898 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.917+0000 7ffbd0636040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.966+0000 7f6f2c9c8640 10 monclient: get_monmap_and_config 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.966+0000 7f6f2c9c8640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.966+0000 7f6f2c9c8640 10 monclient: monmap: 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:02.967793+0000 2024-03-29T10:52:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:02.967793+0000 2024-03-29T10:52:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:02.949 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.967+0000 7f6f2c9c8640 1 Processor -- start 2024-03-29T10:52:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.967+0000 7f6f2c9c8640 1 -- start start 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.967+0000 7f6f2c9c8640 10 monclient: init 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6f280b1780 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6f2813d7e0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6f28143be0 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f28070b30 con 0x7f6f28143be0 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f28070da0 con 0x7f6f2813d7e0 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2809adb0 con 0x7f6f280b1780 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 10 monclient(hunting): authenticate will time out at 805.274512s 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f25d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6f2813d7e0 0x7f6f281434d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43540/0 (socket says 172.21.15.116:43540) 2024-03-29T10:52:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f25d9b640 1 -- 172.21.15.116:0/357037498 learned_addr learned my addr 172.21.15.116:0/357037498 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697917346 0 0) 0x7f6f28070da0 con 0x7f6f2813d7e0 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f14003520 con 0x7f6f2813d7e0 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819500507 0 0) 0x7f6f28070b30 con 0x7f6f28143be0 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f28070da0 con 0x7f6f28143be0 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594767109 0 0) 0x7f6f2809adb0 con 0x7f6f280b1780 2024-03-29T10:52:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f28070b30 con 0x7f6f280b1780 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885114653 0 0) 0x7f6f14003520 con 0x7f6f2813d7e0 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 >> v1:172.21.15.45:6790/0 conn(0x7f6f280b1780 legacy=0x7f6f2813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 >> v1:172.21.15.45:6789/0 conn(0x7f6f28143be0 legacy=0x7f6f2814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 monclient: found mon.noname-c 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28144070 con 0x7f6f2813d7e0 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6f10001690 con 0x7f6f2813d7e0 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2c9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.968+0000 7f6f2559a640 10 monclient: dump: 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f10001870 con 0x7f6f2813d7e0 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2559a640 1 -- 172.21.15.116:0/357037498 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6f10001c80 con 0x7f6f2813d7e0 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 4 set_mon_vals no callback set 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2559a640 10 monclient: dump: 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 10 monclient: shutdown 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/357037498 >> v1:172.21.15.116:6789/0 conn(0x7f6f2813d7e0 legacy=0x7f6f281434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/357037498 shutdown_connections 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/357037498 >> 172.21.15.116:0/357037498 conn(0x7f6f280b58f0 msgr2=0x7f6f280b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/357037498 shutdown_connections 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.969+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/357037498 wait complete. 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.970+0000 7f6f2c9c8640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.970+0000 7f6f2c9c8640 10 monclient: monmap: 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.970+0000 7f6f2c9c8640 1 librados: starting msgr at 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.970+0000 7f6f2c9c8640 1 librados: starting objecter 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.970+0000 7f6f2c9c8640 1 Processor -- start 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 1 -- start start 2024-03-29T10:52:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 1 librados: setting wanted keys 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 1 librados: calling monclient init 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient: init 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6f280b1780 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6f2813d7e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6f28143be0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f28139210 con 0x7f6f2813d7e0 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2813a410 con 0x7f6f28143be0 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f281396a0 con 0x7f6f280b1780 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f2c9c8640 10 monclient(hunting): authenticate will time out at 805.277786s 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f26d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6f28143be0 0x7f6f281e1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43550/0 (socket says 172.21.15.116:43550) 2024-03-29T10:52:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f26d9d640 1 -- 172.21.15.116:0/1151134582 learned_addr learned my addr 172.21.15.116:0/1151134582 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 client.?.objecter ms_handle_connect 0x7f6f28143be0 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 client.?.objecter ms_handle_connect 0x7f6f280b1780 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 client.?.objecter ms_handle_connect 0x7f6f2813d7e0 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826251260 0 0) 0x7f6f2813a410 con 0x7f6f28143be0 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f08004eb0 con 0x7f6f28143be0 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340285122 0 0) 0x7f6f281396a0 con 0x7f6f280b1780 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.971+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f2813a410 con 0x7f6f280b1780 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753602857 0 0) 0x7f6f28139210 con 0x7f6f2813d7e0 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f281396a0 con 0x7f6f2813d7e0 2024-03-29T10:52:02.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748062425 0 0) 0x7f6f08004eb0 con 0x7f6f28143be0 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f28139210 con 0x7f6f28143be0 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6f20001c00 con 0x7f6f28143be0 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.969 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3330873591 0 0) 0x7f6f2813a410 con 0x7f6f280b1780 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f08004eb0 con 0x7f6f280b1780 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6f18001870 con 0x7f6f280b1780 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747502218 0 0) 0x7f6f281396a0 con 0x7f6f2813d7e0 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f2813a410 con 0x7f6f2813d7e0 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6f10001b80 con 0x7f6f2813d7e0 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient(hunting): dump: 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3396810055 0 0) 0x7f6f28139210 con 0x7f6f28143be0 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 >> v1:172.21.15.45:6790/0 conn(0x7f6f280b1780 legacy=0x7f6f28138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 >> v1:172.21.15.45:6789/0 conn(0x7f6f2813d7e0 legacy=0x7f6f281d7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 monclient: found mon.b 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28143420 con 0x7f6f28143be0 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 10 monclient: _renew_subs 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f28138bd0 con 0x7f6f28143be0 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 10 monclient: _renew_subs 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f281d7a80 con 0x7f6f28143be0 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f2c9c8640 1 librados: init done 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f20003990 con 0x7f6f28143be0 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6f20001d60 con 0x7f6f28143be0 2024-03-29T10:52:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 monclient: dump: 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f24d99640 4 set_mon_vals no callback set 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6f20004170 con 0x7f6f28143be0 2024-03-29T10:52:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1332291630 0 0) 0x7f6f20005b40 con 0x7f6f28143be0 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 10 client.4135.objecter ms_dispatch 0x7f6f28000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.972+0000 7f6f1effd640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.973+0000 7f6f2c9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.973+0000 7f6f2c9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.973+0000 7f6f2c9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.973+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f28067c00 con 0x7f6f28143be0 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.975+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6f20005ea0 con 0x7f6f28143be0 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.975+0000 7f6f1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:02.975+0000 7f6f1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:02.976 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:02.991+0000 7ffbd0636040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:52:02.976 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:02.980+0000 7fc21ca3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:52:03.009 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.028+0000 7fc21ca3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:52:03.021 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.039+0000 7ffbd0636040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:52:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.047+0000 7f6f2c9c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3", "id": 4}] 2024-03-29T10:52:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.047+0000 7f6f2c9c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3", "id": 4}] 2024-03-29T10:52:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.047+0000 7f6f2c9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.047+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3", "id": 4} v 0) v1 -- 0x7f6f2804f730 con 0x7f6f28143be0 2024-03-29T10:52:03.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.050+0000 7f6f1effd640 1 -- 172.21.15.116:0/1151134582 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2854940514 0 1697012583) 0x7f6f20004420 con 0x7f6f28143be0 2024-03-29T10:52:03.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3", "id": 4}] 2024-03-29T10:52:03.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:03.032 INFO:teuthology.orchestra.run.smithi116.stdout:4 2024-03-29T10:52:03.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 10 librados: watch_flush enter 2024-03-29T10:52:03.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 10 librados: watch_flush exit 2024-03-29T10:52:03.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 10 monclient: shutdown 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 >> v1:172.21.15.116:6789/0 conn(0x7f6f28143be0 legacy=0x7f6f281e1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 shutdown_connections 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 >> 172.21.15.116:0/1151134582 conn(0x7f6f280b58f0 msgr2=0x7f6f28069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 shutdown_connections 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.051+0000 7f6f2c9c8640 1 -- 172.21.15.116:0/1151134582 wait complete. 2024-03-29T10:52:03.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.052+0000 7f6f2c9c8640 1 librados: shutdown 2024-03-29T10:52:03.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new c5645e2a-7d55-44f9-946e-18a6d8dd20d5 5 2024-03-29T10:52:03.064 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.083+0000 7fc21ca3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:52:03.077 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.096+0000 7ffbd0636040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:52:03.112 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.131+0000 7fc21ca3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:52:03.126 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.145+0000 7ffbd0636040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:52:03.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.161+0000 7f29e1ec8640 10 monclient: get_monmap_and_config 2024-03-29T10:52:03.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.161+0000 7f29e1ec8640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.161+0000 7f29e1ec8640 10 monclient: monmap: 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:03.162446+0000 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:03.162446+0000 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:03.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 1 Processor -- start 2024-03-29T10:52:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 1 -- start start 2024-03-29T10:52:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 10 monclient: init 2024-03-29T10:52:03.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:03.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 10 monclient(hunting): picked mon.noname-b con 0x7f29dc09ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 10 monclient(hunting): picked mon.noname-c con 0x7f29dc09c870 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 10 monclient(hunting): picked mon.noname-a con 0x7f29dc0b4360 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.162+0000 7f29e1ec8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc09dec0 con 0x7f29dc0b4360 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc09da30 con 0x7f29dc09c870 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc0b39c0 con 0x7f29dc09ac40 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 10 monclient(hunting): authenticate will time out at 805.469190s 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29daffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f29dc09c870 0x7f29dc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43564/0 (socket says 172.21.15.116:43564) 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29daffd640 1 -- 172.21.15.116:0/3356050519 learned_addr learned my addr 172.21.15.116:0/3356050519 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128572609 0 0) 0x7f29dc09da30 con 0x7f29dc09c870 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c8003520 con 0x7f29dc09c870 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763302742 0 0) 0x7f29d4001490 con 0x7f29dc0b4360 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29dc09da30 con 0x7f29dc0b4360 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146730246 0 0) 0x7f29dc0b39c0 con 0x7f29dc09ac40 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d4001490 con 0x7f29dc09ac40 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2379358395 0 0) 0x7f29c8003520 con 0x7f29dc09c870 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 >> v1:172.21.15.45:6790/0 conn(0x7f29dc09ac40 legacy=0x7f29dc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 >> v1:172.21.15.45:6789/0 conn(0x7f29dc0b4360 legacy=0x7f29dc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 monclient: found mon.noname-c 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc0b5160 con 0x7f29dc09c870 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f29c4001690 con 0x7f29dc09c870 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: dump: 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29c4001870 con 0x7f29dc09c870 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 1 -- 172.21.15.116:0/3356050519 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f29c4001c80 con 0x7f29dc09c870 2024-03-29T10:52:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29da7fc640 10 monclient: dump: 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.163+0000 7f29e1ec8640 4 set_mon_vals no callback set 2024-03-29T10:52:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 10 monclient: shutdown 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 1 -- 172.21.15.116:0/3356050519 >> v1:172.21.15.116:6789/0 conn(0x7f29dc09c870 legacy=0x7f29dc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 1 -- 172.21.15.116:0/3356050519 shutdown_connections 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 1 -- 172.21.15.116:0/3356050519 >> 172.21.15.116:0/3356050519 conn(0x7f29dc066b00 msgr2=0x7f29dc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 1 -- 172.21.15.116:0/3356050519 shutdown_connections 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.164+0000 7f29e1ec8640 1 -- 172.21.15.116:0/3356050519 wait complete. 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 10 monclient: monmap: 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:03.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 1 librados: starting msgr at 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 1 librados: starting objecter 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 1 Processor -- start 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 1 -- start start 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 1 librados: setting wanted keys 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 1 librados: calling monclient init 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.165+0000 7f29e1ec8640 10 monclient: init 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient(hunting): picked mon.noname-c con 0x7f29dc09ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient(hunting): picked mon.noname-a con 0x7f29dc0b6b00 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient(hunting): picked mon.noname-b con 0x7f29dc1406b0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc1d7e20 con 0x7f29dc0b6b00 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc140b00 con 0x7f29dc1406b0 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc0ad080 con 0x7f29dc09ac40 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29e1ec8640 10 monclient(hunting): authenticate will time out at 805.472648s 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29dbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f29dc1406b0 0x7f29dc1e1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43566/0 (socket says 172.21.15.116:43566) 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29dbfff640 1 -- 172.21.15.116:0/955603421 learned_addr learned my addr 172.21.15.116:0/955603421 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 client.?.objecter ms_handle_connect 0x7f29dc1406b0 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 client.?.objecter ms_handle_connect 0x7f29dc0b6b00 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 client.?.objecter ms_handle_connect 0x7f29dc09ac40 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35609423 0 0) 0x7f29dc140b00 con 0x7f29dc1406b0 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29bc004eb0 con 0x7f29dc1406b0 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908407168 0 0) 0x7f29dc1d7e20 con 0x7f29dc0b6b00 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29dc140b00 con 0x7f29dc0b6b00 2024-03-29T10:52:03.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830727310 0 0) 0x7f29dc0ad080 con 0x7f29dc09ac40 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.166+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29dc1d7e20 con 0x7f29dc09ac40 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889581444 0 0) 0x7f29bc004eb0 con 0x7f29dc1406b0 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29dc0ad080 con 0x7f29dc1406b0 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f29d4001bf0 con 0x7f29dc1406b0 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): dump: 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.166 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2225741932 0 0) 0x7f29dc140b00 con 0x7f29dc0b6b00 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29bc004eb0 con 0x7f29dc0b6b00 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f29c4001b80 con 0x7f29dc0b6b00 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): dump: 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3421771260 0 0) 0x7f29dc1d7e20 con 0x7f29dc09ac40 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29dc140b00 con 0x7f29dc09ac40 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f29cc001830 con 0x7f29dc09ac40 2024-03-29T10:52:03.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient(hunting): dump: 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 31744528 0 0) 0x7f29dc0ad080 con 0x7f29dc1406b0 2024-03-29T10:52:03.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 >> v1:172.21.15.45:6790/0 conn(0x7f29dc09ac40 legacy=0x7f29dc1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 >> v1:172.21.15.45:6789/0 conn(0x7f29dc0b6b00 legacy=0x7f29dc13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 monclient: found mon.b 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc1d7ab0 con 0x7f29dc1406b0 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 10 monclient: _renew_subs 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc1e3570 con 0x7f29dc1406b0 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 10 monclient: _renew_subs 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc1e38e0 con 0x7f29dc1406b0 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29e1ec8640 1 librados: init done 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29d4003990 con 0x7f29dc1406b0 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f29d4003d20 con 0x7f29dc1406b0 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d9ffb640 4 set_mon_vals no callback set 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 monclient: dump: 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f29d40042d0 con 0x7f29dc1406b0 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 4215952827 0 0) 0x7f29d4006ab0 con 0x7f29dc1406b0 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 10 client.4141.objecter ms_dispatch 0x7f29dc000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:52:03.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.167+0000 7f29d3fff640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.168+0000 7f29e1ec8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.168+0000 7f29e1ec8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.168+0000 7f29e1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.168+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c0005080 con 0x7f29dc1406b0 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.170+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f29d4004760 con 0x7f29dc1406b0 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.170+0000 7f29d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.170+0000 7f29d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.242+0000 7f29e1ec8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c5645e2a-7d55-44f9-946e-18a6d8dd20d5", "id": 5}] 2024-03-29T10:52:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.242+0000 7f29e1ec8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c5645e2a-7d55-44f9-946e-18a6d8dd20d5", "id": 5}] 2024-03-29T10:52:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.242+0000 7f29e1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.242+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c5645e2a-7d55-44f9-946e-18a6d8dd20d5", "id": 5} v 0) v1 -- 0x7f29c00058b0 con 0x7f29dc1406b0 2024-03-29T10:52:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29d3fff640 1 -- 172.21.15.116:0/955603421 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5645e2a-7d55-44f9-946e-18a6d8dd20d5", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3087318571 0 1988412176) 0x7f29d4006530 con 0x7f29dc1406b0 2024-03-29T10:52:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c5645e2a-7d55-44f9-946e-18a6d8dd20d5", "id": 5}] 2024-03-29T10:52:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:03.227 INFO:teuthology.orchestra.run.smithi116.stdout:5 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 10 librados: watch_flush enter 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 10 librados: watch_flush exit 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 10 monclient: shutdown 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.246+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 >> v1:172.21.15.116:6789/0 conn(0x7f29dc1406b0 legacy=0x7f29dc1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.247+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 shutdown_connections 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.247+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 >> 172.21.15.116:0/955603421 conn(0x7f29dc066b00 msgr2=0x7f29dc0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.247+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 shutdown_connections 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.247+0000 7f29e1ec8640 1 -- 172.21.15.116:0/955603421 wait complete. 2024-03-29T10:52:03.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.247+0000 7f29e1ec8640 1 librados: shutdown 2024-03-29T10:52:03.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 4cf69c62-224b-49d1-8c83-cf9cec6104af 6 2024-03-29T10:52:03.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.397+0000 7efdcb9af640 10 monclient: get_monmap_and_config 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.397+0000 7efdcb9af640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.397+0000 7efdcb9af640 10 monclient: monmap: 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:03.398142+0000 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:03.398142+0000 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.397+0000 7efdcb9af640 1 Processor -- start 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 1 -- start start 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient: init 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient(hunting): picked mon.noname-b con 0x7efdc409ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient(hunting): picked mon.noname-c con 0x7efdc409c870 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient(hunting): picked mon.noname-a con 0x7efdc40b4360 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc409dec0 con 0x7efdc40b4360 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc409da30 con 0x7efdc409c870 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc40b39c0 con 0x7efdc409ac40 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdcb9af640 10 monclient(hunting): authenticate will time out at 805.704813s 2024-03-29T10:52:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdc8f4a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efdc409c870 0x7efdc413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43574/0 (socket says 172.21.15.116:43574) 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.398+0000 7efdc8f4a640 1 -- 172.21.15.116:0/1826521157 learned_addr learned my addr 172.21.15.116:0/1826521157 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761451983 0 0) 0x7efdc409da30 con 0x7efdc409c870 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb0003520 con 0x7efdc409c870 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67160820 0 0) 0x7efdc40b39c0 con 0x7efdc409ac40 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc409da30 con 0x7efdc409ac40 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612074350 0 0) 0x7efdbc0013c0 con 0x7efdc40b4360 2024-03-29T10:52:03.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc40b39c0 con 0x7efdc40b4360 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650288231 0 0) 0x7efdb0003520 con 0x7efdc409c870 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 >> v1:172.21.15.45:6790/0 conn(0x7efdc409ac40 legacy=0x7efdc40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 >> v1:172.21.15.45:6789/0 conn(0x7efdc40b4360 legacy=0x7efdc414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 monclient: found mon.noname-c 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc40b5160 con 0x7efdc409c870 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efdc0001690 con 0x7efdc409c870 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdcb9af640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdcb9af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: dump: 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdcb9af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc0001870 con 0x7efdc409c870 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 1 -- 172.21.15.116:0/1826521157 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efdc0001c80 con 0x7efdc409c870 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdbbfff640 10 monclient: dump: 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.392 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdcb9af640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.399+0000 7efdcb9af640 4 set_mon_vals no callback set 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 10 monclient: shutdown 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 1 -- 172.21.15.116:0/1826521157 >> v1:172.21.15.116:6789/0 conn(0x7efdc409c870 legacy=0x7efdc413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 1 -- 172.21.15.116:0/1826521157 shutdown_connections 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 1 -- 172.21.15.116:0/1826521157 >> 172.21.15.116:0/1826521157 conn(0x7efdc4066b00 msgr2=0x7efdc4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 1 -- 172.21.15.116:0/1826521157 shutdown_connections 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 1 -- 172.21.15.116:0/1826521157 wait complete. 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.400+0000 7efdcb9af640 10 monclient: monmap: 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 librados: starting msgr at 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 librados: starting objecter 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 Processor -- start 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 -- start start 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 librados: setting wanted keys 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 librados: calling monclient init 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient: init 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient(hunting): picked mon.noname-c con 0x7efdc409ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient(hunting): picked mon.noname-a con 0x7efdc40acf60 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient(hunting): picked mon.noname-b con 0x7efdc40b0800 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc41da7b0 con 0x7efdc40acf60 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc40b0c70 con 0x7efdc40b0800 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc41db350 con 0x7efdc409ac40 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdcb9af640 10 monclient(hunting): authenticate will time out at 805.707930s 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdc9f4c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efdc40b0800 0x7efdc41e1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43578/0 (socket says 172.21.15.116:43578) 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdc9f4c640 1 -- 172.21.15.116:0/3354576992 learned_addr learned my addr 172.21.15.116:0/3354576992 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdb9ffb640 10 client.?.objecter ms_handle_connect 0x7efdc40b0800 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.401+0000 7efdb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 client.?.objecter ms_handle_connect 0x7efdc40acf60 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 client.?.objecter ms_handle_connect 0x7efdc409ac40 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521676603 0 0) 0x7efdc40b0c70 con 0x7efdc40b0800 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda8004eb0 con 0x7efdc40b0800 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295948366 0 0) 0x7efdc41da7b0 con 0x7efdc40acf60 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc40b0c70 con 0x7efdc40acf60 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460635153 0 0) 0x7efdc41db350 con 0x7efdc409ac40 2024-03-29T10:52:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc41da7b0 con 0x7efdc409ac40 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3458729308 0 0) 0x7efda8004eb0 con 0x7efdc40b0800 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc41db350 con 0x7efdc40b0800 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efdbc004130 con 0x7efdc40b0800 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130039729 0 0) 0x7efdc40b0c70 con 0x7efdc40acf60 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda8004eb0 con 0x7efdc40acf60 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efdc0001b80 con 0x7efdc40acf60 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2003111292 0 0) 0x7efdc41da7b0 con 0x7efdc409ac40 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc40b0c70 con 0x7efdc409ac40 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efdb40018b0 con 0x7efdc409ac40 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3040544780 0 0) 0x7efdc41db350 con 0x7efdc40b0800 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 >> v1:172.21.15.45:6790/0 conn(0x7efdc409ac40 legacy=0x7efdc40af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 >> v1:172.21.15.45:6789/0 conn(0x7efdc40acf60 legacy=0x7efdc41dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 monclient: found mon.b 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc41db4e0 con 0x7efdc40b0800 2024-03-29T10:52:03.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 10 monclient: _renew_subs 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc41e3240 con 0x7efdc40b0800 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdbc0046a0 con 0x7efdc40b0800 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efdbc001250 con 0x7efdc40b0800 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 monclient: dump: 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efdbc0017c0 con 0x7efdc40b0800 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdbb7fe640 4 set_mon_vals no callback set 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 10 monclient: _renew_subs 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc41e35b0 con 0x7efdc40b0800 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdcb9af640 1 librados: init done 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 417387399 0 0) 0x7efdbc005b40 con 0x7efdc40b0800 2024-03-29T10:52:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.402+0000 7efdb9ffb640 10 client.4147.objecter ms_dispatch 0x7efdc4000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdb9ffb640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdb9ffb640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdb9ffb640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdcb9af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdcb9af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdcb9af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.403+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc409b020 con 0x7efdc40b0800 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.405+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efdbc001ca0 con 0x7efdc40b0800 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.405+0000 7efdb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.405+0000 7efdb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:03.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.478+0000 7efdcb9af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4cf69c62-224b-49d1-8c83-cf9cec6104af", "id": 6}] 2024-03-29T10:52:03.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.478+0000 7efdcb9af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4cf69c62-224b-49d1-8c83-cf9cec6104af", "id": 6}] 2024-03-29T10:52:03.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.478+0000 7efdcb9af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.478+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4cf69c62-224b-49d1-8c83-cf9cec6104af", "id": 6} v 0) v1 -- 0x7efdc404f730 con 0x7efdc40b0800 2024-03-29T10:52:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.482+0000 7efdb9ffb640 1 -- 172.21.15.116:0/3354576992 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4cf69c62-224b-49d1-8c83-cf9cec6104af", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4130194424 0 1113811849) 0x7efdbc001e40 con 0x7efdc40b0800 2024-03-29T10:52:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.482+0000 7efdb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4cf69c62-224b-49d1-8c83-cf9cec6104af", "id": 6}] 2024-03-29T10:52:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.482+0000 7efdb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:03.464 INFO:teuthology.orchestra.run.smithi116.stdout:6 2024-03-29T10:52:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 10 librados: watch_flush enter 2024-03-29T10:52:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 10 librados: watch_flush exit 2024-03-29T10:52:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 10 monclient: shutdown 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 >> v1:172.21.15.116:6789/0 conn(0x7efdc40b0800 legacy=0x7efdc41e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 shutdown_connections 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 >> 172.21.15.116:0/3354576992 conn(0x7efdc4066b00 msgr2=0x7efdc40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 shutdown_connections 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 1 -- 172.21.15.116:0/3354576992 wait complete. 2024-03-29T10:52:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.483+0000 7efdcb9af640 1 librados: shutdown 2024-03-29T10:52:03.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 8fa620d6-caba-4277-bb1c-aa078e0fac96 7 2024-03-29T10:52:03.510 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.528+0000 7fc21ca3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:52:03.530 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.548+0000 7ffbd0636040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.593+0000 7fb9b3d24640 10 monclient: get_monmap_and_config 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.593+0000 7fb9b3d24640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.593+0000 7fb9b3d24640 10 monclient: monmap: 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:52:03.594656+0000 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:52:03.594656+0000 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:03.576 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.594+0000 7fb9b3d24640 1 Processor -- start 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.594+0000 7fb9b3d24640 1 -- start start 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.594+0000 7fb9b3d24640 10 monclient: init 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac09ac40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac09c870 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac0b4360 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac09dec0 con 0x7fb9ac0b4360 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac09da30 con 0x7fb9ac09c870 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac0b39c0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient(hunting): authenticate will time out at 805.901280s 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b12bf640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb9ac09c870 0x7fb9ac13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43594/0 (socket says 172.21.15.116:43594) 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b12bf640 1 -- 172.21.15.116:0/888077886 learned_addr learned my addr 172.21.15.116:0/888077886 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933299171 0 0) 0x7fb9ac09da30 con 0x7fb9ac09c870 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb998003520 con 0x7fb9ac09c870 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523521262 0 0) 0x7fb9a4001490 con 0x7fb9ac0b4360 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac09da30 con 0x7fb9ac0b4360 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147132200 0 0) 0x7fb9ac0b39c0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4001490 con 0x7fb9ac09ac40 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567991008 0 0) 0x7fb998003520 con 0x7fb9ac09c870 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 >> v1:172.21.15.45:6790/0 conn(0x7fb9ac09ac40 legacy=0x7fb9ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 >> v1:172.21.15.45:6789/0 conn(0x7fb9ac0b4360 legacy=0x7fb9ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 monclient: found mon.noname-c 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0b5160 con 0x7fb9ac09c870 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb99c001690 con 0x7fb9ac09c870 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: dump: 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb99c001870 con 0x7fb9ac09c870 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 1 -- 172.21.15.116:0/888077886 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb99c001c80 con 0x7fb9ac09c870 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b3d24640 4 set_mon_vals no callback set 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.595+0000 7fb9b0abe640 10 monclient: dump: 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 10 monclient: shutdown 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 1 -- 172.21.15.116:0/888077886 >> v1:172.21.15.116:6789/0 conn(0x7fb9ac09c870 legacy=0x7fb9ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 1 -- 172.21.15.116:0/888077886 shutdown_connections 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 1 -- 172.21.15.116:0/888077886 >> 172.21.15.116:0/888077886 conn(0x7fb9ac066b00 msgr2=0x7fb9ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 1 -- 172.21.15.116:0/888077886 shutdown_connections 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.596+0000 7fb9b3d24640 1 -- 172.21.15.116:0/888077886 wait complete. 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 10 monclient: monmap: 2024-03-29T10:52:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 1 librados: starting msgr at 2024-03-29T10:52:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 1 librados: starting objecter 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 1 Processor -- start 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 1 -- start start 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 1 librados: setting wanted keys 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 1 librados: calling monclient init 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 10 monclient: init 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.597+0000 7fb9b3d24640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac09ac40 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac0b6b00 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac1406b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac1d7e20 con 0x7fb9ac0b6b00 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac140b00 con 0x7fb9ac09ac40 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac0ad080 con 0x7fb9ac1406b0 2024-03-29T10:52:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient(hunting): authenticate will time out at 805.904214s 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b1ac0640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb9ac09ac40 0x7fb9ac1d73a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43596/0 (socket says 172.21.15.116:43596) 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b1ac0640 1 -- 172.21.15.116:0/2973977292 learned_addr learned my addr 172.21.15.116:0/2973977292 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 client.?.objecter ms_handle_connect 0x7fb9ac09ac40 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 client.?.objecter ms_handle_connect 0x7fb9ac1406b0 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 client.?.objecter ms_handle_connect 0x7fb9ac0b6b00 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386364613 0 0) 0x7fb9ac140b00 con 0x7fb9ac09ac40 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990004eb0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006901005 0 0) 0x7fb9ac0ad080 con 0x7fb9ac1406b0 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac140b00 con 0x7fb9ac1406b0 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999338708 0 0) 0x7fb9ac1d7e20 con 0x7fb9ac0b6b00 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac0ad080 con 0x7fb9ac0b6b00 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2346043359 0 0) 0x7fb990004eb0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ac1d7e20 con 0x7fb9ac09ac40 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb9a8001930 con 0x7fb9ac09ac40 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): dump: 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 955854486 0 0) 0x7fb9ac140b00 con 0x7fb9ac1406b0 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb990004eb0 con 0x7fb9ac1406b0 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb9a4001b40 con 0x7fb9ac1406b0 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): dump: 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1823559269 0 0) 0x7fb9ac0ad080 con 0x7fb9ac0b6b00 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ac140b00 con 0x7fb9ac0b6b00 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb99c001b80 con 0x7fb9ac0b6b00 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient(hunting): dump: 2024-03-29T10:52:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 841890734 0 0) 0x7fb9ac1d7e20 con 0x7fb9ac09ac40 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 >> v1:172.21.15.45:6790/0 conn(0x7fb9ac1406b0 legacy=0x7fb9ac1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 >> v1:172.21.15.45:6789/0 conn(0x7fb9ac0b6b00 legacy=0x7fb9ac13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 monclient: found mon.b 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac1d7ab0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:52:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient: _renew_subs 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac1e3570 con 0x7fb9ac09ac40 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient: _renew_subs 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac1e38e0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9b3d24640 1 librados: init done 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9a8003610 con 0x7fb9ac09ac40 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb9a80039e0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a3fff640 4 set_mon_vals no callback set 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.598+0000 7fb9a27fc640 10 monclient: dump: 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb9a8003f90 con 0x7fb9ac09ac40 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1165338601 0 0) 0x7fb9a8006860 con 0x7fb9ac09ac40 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9a27fc640 10 client.4153.objecter ms_dispatch 0x7fb9ac000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9a27fc640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9a27fc640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9a27fc640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9b3d24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9b3d24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9b3d24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.599+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb994005080 con 0x7fb9ac09ac40 2024-03-29T10:52:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.601+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb9a80043d0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.601+0000 7fb9a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.601+0000 7fb9a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:03.650 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.669+0000 7fc21ca3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:52:03.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.673+0000 7fb9b3d24640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8fa620d6-caba-4277-bb1c-aa078e0fac96", "id": 7}] 2024-03-29T10:52:03.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.673+0000 7fb9b3d24640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8fa620d6-caba-4277-bb1c-aa078e0fac96", "id": 7}] 2024-03-29T10:52:03.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.673+0000 7fb9b3d24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:03.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.673+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8fa620d6-caba-4277-bb1c-aa078e0fac96", "id": 7} v 0) v1 -- 0x7fb9940058b0 con 0x7fb9ac09ac40 2024-03-29T10:52:03.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.676+0000 7fb9a27fc640 1 -- 172.21.15.116:0/2973977292 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8fa620d6-caba-4277-bb1c-aa078e0fac96", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1927526979 0 1371665406) 0x7fb9a8006a70 con 0x7fb9ac09ac40 2024-03-29T10:52:03.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.676+0000 7fb9a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8fa620d6-caba-4277-bb1c-aa078e0fac96", "id": 7}] 2024-03-29T10:52:03.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.676+0000 7fb9a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:03.658 INFO:teuthology.orchestra.run.smithi116.stdout:7 2024-03-29T10:52:03.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 10 librados: watch_flush enter 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 10 librados: watch_flush exit 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 10 monclient: shutdown 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 >> v1:172.21.15.116:6789/0 conn(0x7fb9ac09ac40 legacy=0x7fb9ac1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 shutdown_connections 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 >> 172.21.15.116:0/2973977292 conn(0x7fb9ac066b00 msgr2=0x7fb9ac0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 shutdown_connections 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 1 -- 172.21.15.116:0/2973977292 wait complete. 2024-03-29T10:52:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-29T10:52:03.677+0000 7fb9b3d24640 1 librados: shutdown 2024-03-29T10:52:03.670 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:52:03.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:52:03.670 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.689+0000 7ffbd0636040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:52:03.672 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:52:03.672 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:52:03.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:52:03.675 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:52:03.675 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:52:03.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:52:03.677 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:52:03.677 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:52:03.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:52:03.679 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:52:03.680 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:52:03.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:52:03.682 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:52:03.682 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:52:03.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:52:03.684 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:52:03.684 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:52:03.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:52:03.686 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:52:03.686 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:52:03.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:52:03.688 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:52:03.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:03.790 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.809+0000 7fc21ca3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:52:03.812 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.830+0000 7ffbd0636040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:52:03.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.834+0000 7fe76f76d640 10 monclient: get_monmap_and_config 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.834+0000 7fe76f76d640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.834+0000 7fe76f76d640 10 monclient: monmap: 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:52:03.836461+0000 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:52:03.836461+0000 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:03.817 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.835+0000 7fe76f76d640 1 Processor -- start 2024-03-29T10:52:03.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.835+0000 7fe76f76d640 1 -- start start 2024-03-29T10:52:03.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.835+0000 7fe76f76d640 10 monclient: init 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.835+0000 7fe76f76d640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.835+0000 7fe76f76d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.835+0000 7fe76f76d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7680b8240 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7680b1f60 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 10 monclient(hunting): picked mon.noname-c con 0x7fe76809c8c0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76d509640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe7680b8240 0x7fe76813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37916/0 (socket says 172.21.15.45:37916) 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76d509640 1 -- 172.21.15.45:0/1892043233 learned_addr learned my addr 172.21.15.45:0/1892043233 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76809bd80 con 0x7fe7680b1f60 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76809b960 con 0x7fe76809c8c0 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680b2ec0 con 0x7fe7680b8240 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 10 monclient(hunting): authenticate will time out at 798.106365s 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287524268 0 0) 0x7fe76809bd80 con 0x7fe7680b1f60 2024-03-29T10:52:03.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe754003520 con 0x7fe7680b1f60 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634961352 0 0) 0x7fe7680b2ec0 con 0x7fe7680b8240 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe76809bd80 con 0x7fe7680b8240 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689804113 0 0) 0x7fe76809b960 con 0x7fe76809c8c0 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7680b2ec0 con 0x7fe76809c8c0 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383240795 0 0) 0x7fe754003520 con 0x7fe7680b1f60 2024-03-29T10:52:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 >> v1:172.21.15.45:6790/0 conn(0x7fe7680b8240 legacy=0x7fe76813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 >> v1:172.21.15.116:6789/0 conn(0x7fe76809c8c0 legacy=0x7fe76814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 monclient: found mon.noname-a 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76813fb60 con 0x7fe7680b1f60 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe758001690 con 0x7fe7680b1f60 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: dump: 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.833 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 5 monclient: authenticate success, global_id 4185 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe758001870 con 0x7fe7680b1f60 2024-03-29T10:52:03.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 1 -- 172.21.15.45:0/1892043233 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe758001c80 con 0x7fe7680b1f60 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe75ffff640 10 monclient: dump: 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.835 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:03.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.836+0000 7fe76f76d640 4 set_mon_vals no callback set 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 10 monclient: shutdown 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 >> v1:172.21.15.45:6789/0 conn(0x7fe7680b1f60 legacy=0x7fe76809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 shutdown_connections 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 >> 172.21.15.45:0/1892043233 conn(0x7fe768066b00 msgr2=0x7fe768066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 shutdown_connections 2024-03-29T10:52:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.837+0000 7fe76f76d640 1 -- 172.21.15.45:0/1892043233 wait complete. 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 10 monclient: monmap: 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.838 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 1 librados: starting msgr at 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 1 librados: starting objecter 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 1 Processor -- start 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 1 -- start start 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 1 librados: setting wanted keys 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 1 librados: calling monclient init 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.838+0000 7fe76f76d640 10 monclient: init 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient(hunting): picked mon.noname-c con 0x7fe76809c8c0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:03.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7680b8240 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient(hunting): picked mon.noname-b con 0x7fe768131f30 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe768132cf0 con 0x7fe7680b8240 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe768132fd0 con 0x7fe768131f30 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76cd08640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe7680b8240 0x7fe768144ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58420/0 (socket says 172.21.15.45:58420) 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76d509640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe76809c8c0 0x7fe7681336d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37918/0 (socket says 172.21.15.45:37918) 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76cd08640 1 -- 172.21.15.45:0/2982124798 learned_addr learned my addr 172.21.15.45:0/2982124798 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7681332b0 con 0x7fe76809c8c0 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 client.?.objecter ms_handle_connect 0x7fe76809c8c0 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 client.?.objecter ms_handle_connect 0x7fe7680b8240 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe76f76d640 10 monclient(hunting): authenticate will time out at 798.109693s 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 client.?.objecter ms_handle_connect 0x7fe768131f30 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451697998 0 0) 0x7fe7681332b0 con 0x7fe76809c8c0 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c004eb0 con 0x7fe76809c8c0 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246606092 0 0) 0x7fe768132cf0 con 0x7fe7680b8240 2024-03-29T10:52:03.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7681332b0 con 0x7fe7680b8240 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562562326 0 0) 0x7fe768132fd0 con 0x7fe768131f30 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.839+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe768132cf0 con 0x7fe768131f30 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529888232 0 0) 0x7fe74c004eb0 con 0x7fe76809c8c0 2024-03-29T10:52:03.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe768132fd0 con 0x7fe76809c8c0 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe764001860 con 0x7fe76809c8c0 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): dump: 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.843 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000059831 0 0) 0x7fe7681332b0 con 0x7fe7680b8240 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe74c004eb0 con 0x7fe7680b8240 2024-03-29T10:52:03.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe758001b80 con 0x7fe7680b8240 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): dump: 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.845 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492959928 0 0) 0x7fe768132cf0 con 0x7fe768131f30 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7681332b0 con 0x7fe768131f30 2024-03-29T10:52:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe760001b50 con 0x7fe768131f30 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient(hunting): dump: 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 507925915 0 0) 0x7fe74c004eb0 con 0x7fe7680b8240 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 >> v1:172.21.15.45:6790/0 conn(0x7fe76809c8c0 legacy=0x7fe7681336d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 >> v1:172.21.15.116:6789/0 conn(0x7fe768131f30 legacy=0x7fe7681e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 monclient: found mon.a 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768130fe0 con 0x7fe7680b8240 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:52:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 10 monclient: _renew_subs 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe768131290 con 0x7fe7680b8240 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe758003960 con 0x7fe7680b8240 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe758003d30 con 0x7fe7680b8240 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 monclient: dump: 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75f7fe640 4 set_mon_vals no callback set 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 10 monclient: _renew_subs 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe7580042e0 con 0x7fe7680b8240 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681e3850 con 0x7fe7680b8240 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe76f76d640 1 librados: init done 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fe758005f10 con 0x7fe7680b8240 2024-03-29T10:52:03.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 10 client.4188.objecter ms_dispatch 0x7fe768000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.840+0000 7fe75dffb640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.841+0000 7fe75dffb640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.841+0000 7fe76f76d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.841+0000 7fe76f76d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.841+0000 7fe76f76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.841+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76809b170 con 0x7fe7680b8240 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.843+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe758005840 con 0x7fe7680b8240 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.843+0000 7fe75dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.843+0000 7fe75dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:03.854 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.857+0000 7fc21ca3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:52:03.857 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.876+0000 7ffbd0636040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:52:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.905+0000 7fe76f76d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.905+0000 7fe76f76d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.905+0000 7fe76f76d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.905+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe76804f730 con 0x7fe7680b8240 2024-03-29T10:52:03.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe75dffb640 1 -- 172.21.15.45:0/2982124798 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fe758005aa0 con 0x7fe7680b8240 2024-03-29T10:52:03.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe75dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:03.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe75dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:03.889 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T10:52:03.889 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:03.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe76f76d640 10 librados: watch_flush enter 2024-03-29T10:52:03.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe76f76d640 10 librados: watch_flush exit 2024-03-29T10:52:03.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe76f76d640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T10:52:03.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.906+0000 7fe76f76d640 10 client.4188.objecter successfully canceled tick 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 10 monclient: shutdown 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 >> v1:172.21.15.45:6789/0 conn(0x7fe7680b8240 legacy=0x7fe768144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 shutdown_connections 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 >> 172.21.15.45:0/2982124798 conn(0x7fe768066b00 msgr2=0x7fe768066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 shutdown_connections 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 1 -- 172.21.15.45:0/2982124798 wait complete. 2024-03-29T10:52:03.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T10:52:03.907+0000 7fe76f76d640 1 librados: shutdown 2024-03-29T10:52:03.900 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:52:03.900 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:52:03.958 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:03.975+0000 7fc21ca3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:52:03.968 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:03.986+0000 7ffbd0636040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:52:04.056 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:04.074+0000 7fc21ca3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:52:04.062 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:04.081+0000 7ffbd0636040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:52:04.216 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:04.234+0000 7ffbd0636040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:52:04.216 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:04.234+0000 7fc21ca3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:52:04.257 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:04.276+0000 7ffbd0636040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:52:04.260 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:04.278+0000 7fc21ca3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:52:04.276 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-03-29T10:52:04.295+0000 7f12e3a17640 -1 Falling back to public interface 2024-03-29T10:52:04.279 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-03-29T10:52:04.298+0000 7ffbac414640 -1 Falling back to public interface 2024-03-29T10:52:04.286 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-03-29T10:52:04.304+0000 7f95477bb640 -1 Falling back to public interface 2024-03-29T10:52:04.299 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-03-29T10:52:04.317+0000 7fdec1e24640 -1 Falling back to public interface 2024-03-29T10:52:04.299 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-03-29T10:52:04.317+0000 7f1a943e1640 -1 Falling back to public interface 2024-03-29T10:52:04.302 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-03-29T10:52:04.320+0000 7f45a22a6640 -1 Falling back to public interface 2024-03-29T10:52:04.312 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-03-29T10:52:04.330+0000 7fc50720f640 -1 Falling back to public interface 2024-03-29T10:52:04.317 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-03-29T10:52:04.335+0000 7fa45fed5640 -1 Falling back to public interface 2024-03-29T10:52:04.415 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:04.434+0000 7ffbd0636040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:52:04.424 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:04.442+0000 7fc21ca3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:52:04.707 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.814+0000 7ff60d78f640 10 monclient: get_monmap_and_config 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.814+0000 7ff60d78f640 10 monclient: build_initial_monmap 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.814+0000 7ff60d78f640 10 monclient: monmap: 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:04.816527+0000 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:04.816527+0000 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.797 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:04.798 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:04.798 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:04.798 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 1 Processor -- start 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 1 -- start start 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient: init 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6081457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:04.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient(hunting): picked mon.noname-c con 0x7ff60813f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6080a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6080aaef0 con 0x7ff6081457f0 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff608145bd0 con 0x7ff60813f650 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff608145eb0 con 0x7ff6080a9d70 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.815+0000 7ff60d78f640 10 monclient(hunting): authenticate will time out at 799.085901s 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff606ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff6081457f0 0x7ff60813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58424/0 (socket says 172.21.15.45:58424) 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff606ffd640 1 -- 172.21.15.45:0/1039204596 learned_addr learned my addr 172.21.15.45:0/1039204596 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848277236 0 0) 0x7ff6080aaef0 con 0x7ff6081457f0 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f0003520 con 0x7ff6081457f0 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399912125 0 0) 0x7ff608145eb0 con 0x7ff6080a9d70 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6080aaef0 con 0x7ff6080a9d70 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168008839 0 0) 0x7ff608145bd0 con 0x7ff60813f650 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff608145eb0 con 0x7ff60813f650 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182174080 0 0) 0x7ff6080aaef0 con 0x7ff6080a9d70 2024-03-29T10:52:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 >> v1:172.21.15.116:6789/0 conn(0x7ff60813f650 legacy=0x7ff6080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 >> v1:172.21.15.45:6789/0 conn(0x7ff6081457f0 legacy=0x7ff60813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 monclient: found mon.noname-b 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff608146340 con 0x7ff6080a9d70 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff60d78f640 5 monclient: authenticate success, global_id 4190 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff6000019f0 con 0x7ff6080a9d70 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff60d78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:04.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: dump: 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff60d78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff600001b50 con 0x7ff6080a9d70 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:04.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 1 -- 172.21.15.45:0/1039204596 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff600001fa0 con 0x7ff6080a9d70 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff60d78f640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff60d78f640 4 set_mon_vals no callback set 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.816+0000 7ff605ffb640 10 monclient: dump: 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:04.810 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 10 monclient: shutdown 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 1 -- 172.21.15.45:0/1039204596 >> v1:172.21.15.45:6790/0 conn(0x7ff6080a9d70 legacy=0x7ff60814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 1 -- 172.21.15.45:0/1039204596 shutdown_connections 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 1 -- 172.21.15.45:0/1039204596 >> 172.21.15.45:0/1039204596 conn(0x7ff608066a00 msgr2=0x7ff608066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 1 -- 172.21.15.45:0/1039204596 shutdown_connections 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 1 -- 172.21.15.45:0/1039204596 wait complete. 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 10 monclient: build_initial_monmap 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.817+0000 7ff60d78f640 10 monclient: monmap: 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:04.811 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 1 librados: starting msgr at 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 1 librados: starting objecter 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 1 Processor -- start 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 1 -- start start 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 1 librados: setting wanted keys 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 1 librados: calling monclient init 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 10 monclient: init 2024-03-29T10:52:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.818+0000 7ff60d78f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6080a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 10 monclient(hunting): picked mon.noname-b con 0x7ff60813f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6081457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6080ada10 con 0x7ff6080a9d70 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6080b1800 con 0x7ff60813f650 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6080abd10 con 0x7ff6081457f0 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff60d78f640 10 monclient(hunting): authenticate will time out at 799.089065s 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff606ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff6080a9d70 0x7ff6080ab600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58430/0 (socket says 172.21.15.45:58430) 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff6077fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff6081457f0 0x7ff6080b10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37942/0 (socket says 172.21.15.45:37942) 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff606ffd640 1 -- 172.21.15.45:0/639739860 learned_addr learned my addr 172.21.15.45:0/639739860 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 client.?.objecter ms_handle_connect 0x7ff6080a9d70 2024-03-29T10:52:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 client.?.objecter ms_handle_connect 0x7ff6081457f0 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 client.?.objecter ms_handle_connect 0x7ff60813f650 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899428733 0 0) 0x7ff6080ada10 con 0x7ff6080a9d70 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f4004eb0 con 0x7ff6080a9d70 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107454709 0 0) 0x7ff6080b1800 con 0x7ff60813f650 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6080ada10 con 0x7ff60813f650 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193065038 0 0) 0x7ff6080abd10 con 0x7ff6081457f0 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6080b1800 con 0x7ff6081457f0 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3884586212 0 0) 0x7ff6080ada10 con 0x7ff60813f650 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6080abd10 con 0x7ff60813f650 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1782358789 0 0) 0x7ff5f4004eb0 con 0x7ff6080a9d70 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6080ada10 con 0x7ff6080a9d70 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff5fc001870 con 0x7ff60813f650 2024-03-29T10:52:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 207650388 0 0) 0x7ff6080b1800 con 0x7ff6081457f0 2024-03-29T10:52:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5f4004eb0 con 0x7ff6081457f0 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff5f8001830 con 0x7ff6080a9d70 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:04.817 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff600001e60 con 0x7ff6081457f0 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1236904724 0 0) 0x7ff6080abd10 con 0x7ff60813f650 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 >> v1:172.21.15.45:6790/0 conn(0x7ff6081457f0 legacy=0x7ff6080b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 >> v1:172.21.15.45:6789/0 conn(0x7ff6080a9d70 legacy=0x7ff6080ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 monclient: found mon.b 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080ad5a0 con 0x7ff60813f650 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.819+0000 7ff5ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 5 monclient: authenticate success, global_id 4189 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 10 monclient: _renew_subs 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6080adea0 con 0x7ff60813f650 2024-03-29T10:52:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 10 monclient: _renew_subs 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6080ad2c0 con 0x7ff60813f650 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff60d78f640 1 librados: init done 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5fc003610 con 0x7ff60813f650 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff5fc003920 con 0x7ff60813f650 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff6057fa640 4 set_mon_vals no callback set 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 10 monclient: dump: 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:04.820 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff5fc003e10 con 0x7ff60813f650 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7ff5fc005630 con 0x7ff60813f650 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 10 client.4189.objecter ms_dispatch 0x7ff608000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 3 client.4189.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 3 client.4189.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.820+0000 7ff5ef7fe640 20 client.4189.objecter dump_active .. 0 homeless 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.821+0000 7ff60d78f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.821+0000 7ff60d78f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.821+0000 7ff60d78f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.821+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e8005080 con 0x7ff60813f650 2024-03-29T10:52:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.823+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff5fc005010 con 0x7ff60813f650 2024-03-29T10:52:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.823+0000 7ff5ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.823+0000 7ff5ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:04.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff60d78f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:04.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff60d78f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:04.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff60d78f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:04.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5e8005330 con 0x7ff60813f650 2024-03-29T10:52:04.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff5ef7fe640 1 -- 172.21.15.45:0/639739860 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7ff5fc005360 con 0x7ff60813f650 2024-03-29T10:52:04.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff5ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:04.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.886+0000 7ff5ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:04.869 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:04.870 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 10 librados: watch_flush enter 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 10 librados: watch_flush exit 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 20 client.4189.objecter shutdown clearing up homeless session... 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 10 client.4189.objecter successfully canceled tick 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 10 monclient: shutdown 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 >> v1:172.21.15.116:6789/0 conn(0x7ff60813f650 legacy=0x7ff6080b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 shutdown_connections 2024-03-29T10:52:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 >> 172.21.15.45:0/639739860 conn(0x7ff608066a00 msgr2=0x7ff608066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 shutdown_connections 2024-03-29T10:52:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 1 -- 172.21.15.45:0/639739860 wait complete. 2024-03-29T10:52:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:04.887+0000 7ff60d78f640 1 librados: shutdown 2024-03-29T10:52:04.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:05.226 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:05.245+0000 7ffbd0636040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:52:05.281 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:05.298+0000 7fc21ca3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:52:05.500 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:05.518+0000 7ffbd0636040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:52:05.550 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:05.569+0000 7ffbd0636040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:52:05.567 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:05.584+0000 7fc21ca3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:52:05.620 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:05.637+0000 7fc21ca3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:52:05.664 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:05.683+0000 7ffbd0636040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:52:05.748 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T10:52:05.766+0000 7fc21ca3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:52:06.678 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:06.697+0000 7ffb71410640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:52:06.679 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-03-29T10:52:06.697+0000 7ffb71410640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:52:07.742 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-03-29T10:52:07.760+0000 7f12e3a17640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:52:07.759 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-03-29T10:52:07.776+0000 7f95477bb640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:52:07.794 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-03-29T10:52:07.813+0000 7ffbac414640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:52:07.807 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-03-29T10:52:07.824+0000 7fdec1e24640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:52:07.817 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-03-29T10:52:07.835+0000 7f45a22a6640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:52:07.828 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-03-29T10:52:07.845+0000 7f1a943e1640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:52:07.846 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-03-29T10:52:07.865+0000 7fa45fed5640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:52:07.862 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-03-29T10:52:07.881+0000 7fc50720f640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:52:11.687 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.859+0000 7f3784a0e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.859+0000 7f3784a0e640 10 monclient: build_initial_monmap 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.859+0000 7f3784a0e640 10 monclient: monmap: 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:11.861351+0000 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:11.861351+0000 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.842 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:11.843 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:11.843 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:11.843 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.860+0000 7f3784a0e640 1 Processor -- start 2024-03-29T10:52:11.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.861+0000 7f3784a0e640 1 -- start start 2024-03-29T10:52:11.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.861+0000 7f3784a0e640 10 monclient: init 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f37801457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f378013f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f37800a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37800aaef0 con 0x7f378013f650 2024-03-29T10:52:11.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3780145bd0 con 0x7f37801457f0 2024-03-29T10:52:11.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3780145eb0 con 0x7f37800a9d70 2024-03-29T10:52:11.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:11.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f3784a0e640 10 monclient(hunting): authenticate will time out at 806.132212s 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f377e59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f37801457f0 0x7f378013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:46292/0 (socket says 172.21.15.45:46292) 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.862+0000 7f377e59c640 1 -- 172.21.15.45:0/2681511764 learned_addr learned my addr 172.21.15.45:0/2681511764 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246586202 0 0) 0x7f3780145bd0 con 0x7f37801457f0 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:52:11.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3768003520 con 0x7f37801457f0 2024-03-29T10:52:11.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655700530 0 0) 0x7f37800aaef0 con 0x7f378013f650 2024-03-29T10:52:11.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient(hunting): my global_id is 4236 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780145bd0 con 0x7f378013f650 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054705668 0 0) 0x7f3768003520 con 0x7f37801457f0 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 >> v1:172.21.15.45:6790/0 conn(0x7f37800a9d70 legacy=0x7f378014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 >> v1:172.21.15.45:6789/0 conn(0x7f378013f650 legacy=0x7f37800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 monclient: found mon.noname-c 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780146340 con 0x7f37801457f0 2024-03-29T10:52:11.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: _finish_auth 0 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3774001690 con 0x7f37801457f0 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f377d59a640 10 monclient: dump: 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.849 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f3784a0e640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.863+0000 7f3784a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:11.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37740017f0 con 0x7f37801457f0 2024-03-29T10:52:11.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f377d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:11.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f377d59a640 1 -- 172.21.15.45:0/2681511764 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3774001c00 con 0x7f37801457f0 2024-03-29T10:52:11.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f377d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f377d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:11.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f377d59a640 10 monclient: dump: 2024-03-29T10:52:11.851 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f3784a0e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:11.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.864+0000 7f3784a0e640 4 set_mon_vals no callback set 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 10 monclient: shutdown 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 1 -- 172.21.15.45:0/2681511764 >> v1:172.21.15.116:6789/0 conn(0x7f37801457f0 legacy=0x7f378013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 1 -- 172.21.15.45:0/2681511764 shutdown_connections 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 1 -- 172.21.15.45:0/2681511764 >> 172.21.15.45:0/2681511764 conn(0x7f3780066a00 msgr2=0x7f3780066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 1 -- 172.21.15.45:0/2681511764 shutdown_connections 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.865+0000 7f3784a0e640 1 -- 172.21.15.45:0/2681511764 wait complete. 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.866+0000 7f3784a0e640 10 monclient: build_initial_monmap 2024-03-29T10:52:11.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.866+0000 7f3784a0e640 10 monclient: monmap: 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 1 librados: starting msgr at 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 1 librados: starting objecter 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 1 Processor -- start 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 1 -- start start 2024-03-29T10:52:11.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 1 librados: setting wanted keys 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 1 librados: calling monclient init 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.867+0000 7f3784a0e640 10 monclient: init 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f37800a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f378013f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f3780137700 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3780142df0 con 0x7f378013f650 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37800ac860 con 0x7f3780137700 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3780141dc0 con 0x7f37800a9d70 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f3784a0e640 10 monclient(hunting): authenticate will time out at 806.138206s 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f377e59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f37800a9d70 0x7f3780136ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59780/0 (socket says 172.21.15.45:59780) 2024-03-29T10:52:11.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f377e59c640 1 -- 172.21.15.45:0/3926711205 learned_addr learned my addr 172.21.15.45:0/3926711205 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f376effd640 10 client.?.objecter ms_handle_connect 0x7f37800a9d70 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f376effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f376effd640 10 client.?.objecter ms_handle_connect 0x7f3780137700 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.868+0000 7f376effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785942279 0 0) 0x7f37800ac860 con 0x7f3780137700 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3760004eb0 con 0x7f3780137700 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 client.?.objecter ms_handle_connect 0x7f378013f650 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770147648 0 0) 0x7f3780141dc0 con 0x7f37800a9d70 2024-03-29T10:52:11.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.869+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37800ac860 con 0x7f37800a9d70 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 148035986 0 0) 0x7f3760004eb0 con 0x7f3780137700 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3780141dc0 con 0x7f3780137700 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3778001890 con 0x7f3780137700 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): dump: 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.867 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924770235 0 0) 0x7f3780142df0 con 0x7f378013f650 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3760004eb0 con 0x7f378013f650 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037590262 0 0) 0x7f37800ac860 con 0x7f37800a9d70 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3760005b30 con 0x7f37800a9d70 2024-03-29T10:52:11.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3774001bc0 con 0x7f37800a9d70 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient(hunting): dump: 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.869 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1145809263 0 0) 0x7f3780141dc0 con 0x7f3780137700 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 >> v1:172.21.15.45:6790/0 conn(0x7f37800a9d70 legacy=0x7f3780136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 >> v1:172.21.15.45:6789/0 conn(0x7f378013f650 legacy=0x7f37801d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 monclient: found mon.b 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780137c10 con 0x7f3780137700 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient: _finish_auth 0 2024-03-29T10:52:11.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.870+0000 7f376effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 5 monclient: authenticate success, global_id 4237 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 10 monclient: _renew_subs 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37800ab600 con 0x7f3780137700 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 10 monclient: _renew_subs 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3780139c40 con 0x7f3780137700 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f3784a0e640 1 librados: init done 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3778001da0 con 0x7f3780137700 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f376effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3778003c10 con 0x7f3780137700 2024-03-29T10:52:11.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f376effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f376effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.871+0000 7f377cd99640 4 set_mon_vals no callback set 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.872+0000 7f376effd640 10 monclient: dump: 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:11.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.873+0000 7f3784a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.874+0000 7f3784a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.874+0000 7f3784a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.874+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3780075f10 con 0x7f3780137700 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f3778019be0 con 0x7f3780137700 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f3778083ab0 con 0x7f3780137700 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f376effd640 10 client.4237.objecter ms_dispatch 0x7f3780000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f376effd640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f376effd640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f377e59c640 10 monclient: get_auth_request con 0x7f376006e1c0 auth_method 0 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.875+0000 7f376effd640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T10:52:11.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.876+0000 7f376effd640 10 client.4237.objecter ms_handle_connect 0x7f376006e1c0 2024-03-29T10:52:11.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.881+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f377804f1c0 con 0x7f3780137700 2024-03-29T10:52:11.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.881+0000 7f376effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:11.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:11.881+0000 7f376effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:12.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.248+0000 7f3784a0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.248+0000 7f3784a0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.248+0000 7f3784a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:12.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.248+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37800b3680 con 0x7f3780137700 2024-03-29T10:52:12.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.249+0000 7f376effd640 1 -- 172.21.15.45:0/3926711205 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f37780512c0 con 0x7f3780137700 2024-03-29T10:52:12.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.249+0000 7f376effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.249+0000 7f376effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:12.232 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:12.232 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:12.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 10 librados: watch_flush enter 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 10 librados: watch_flush exit 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 10 client.4237.objecter successfully canceled tick 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 >> v1:172.21.15.116:6816/3290579615 conn(0x7f376006e1c0 legacy=0x7f3760070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 10 monclient: shutdown 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 >> v1:172.21.15.116:6789/0 conn(0x7f3780137700 legacy=0x7f37801e2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 shutdown_connections 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 >> 172.21.15.45:0/3926711205 conn(0x7f3780066a00 msgr2=0x7f37800b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:12.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.253+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 shutdown_connections 2024-03-29T10:52:12.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.254+0000 7f3784a0e640 1 -- 172.21.15.45:0/3926711205 wait complete. 2024-03-29T10:52:12.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:12.254+0000 7f3784a0e640 1 librados: shutdown 2024-03-29T10:52:12.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:19.058 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.234+0000 7fe2a8031640 10 monclient: get_monmap_and_config 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.234+0000 7fe2a8031640 10 monclient: build_initial_monmap 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.234+0000 7fe2a8031640 10 monclient: monmap: 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:19.235672+0000 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:19.235672+0000 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:19.216 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.235+0000 7fe2a8031640 1 Processor -- start 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.235+0000 7fe2a8031640 1 -- start start 2024-03-29T10:52:19.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.235+0000 7fe2a8031640 10 monclient: init 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.236+0000 7fe2a8031640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.236+0000 7fe2a8031640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.236+0000 7fe2a8031640 10 monclient(hunting): picked mon.noname-c con 0x7fe2a01457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 10 monclient(hunting): picked mon.noname-a con 0x7fe2a013f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 10 monclient(hunting): picked mon.noname-b con 0x7fe2a00a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a00aaef0 con 0x7fe2a013f650 2024-03-29T10:52:19.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a0145bd0 con 0x7fe2a01457f0 2024-03-29T10:52:19.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a0145eb0 con 0x7fe2a00a9d70 2024-03-29T10:52:19.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:19.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a5dcd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe2a01457f0 0x7fe2a013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:34322/0 (socket says 172.21.15.45:34322) 2024-03-29T10:52:19.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a5dcd640 1 -- 172.21.15.45:0/2172014059 learned_addr learned my addr 172.21.15.45:0/2172014059 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:19.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a8031640 10 monclient(hunting): authenticate will time out at 813.506221s 2024-03-29T10:52:19.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175038268 0 0) 0x7fe2a00aaef0 con 0x7fe2a013f650 2024-03-29T10:52:19.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a4dcb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a4dcb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a4dcb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a4dcb640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.237+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c003520 con 0x7fe2a013f650 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597220903 0 0) 0x7fe2a0145eb0 con 0x7fe2a00a9d70 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a00aaef0 con 0x7fe2a00a9d70 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1055718062 0 0) 0x7fe28c003520 con 0x7fe2a013f650 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 >> v1:172.21.15.45:6790/0 conn(0x7fe2a00a9d70 legacy=0x7fe2a014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 >> v1:172.21.15.116:6789/0 conn(0x7fe2a01457f0 legacy=0x7fe2a013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 monclient: found mon.noname-a 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a0146340 con 0x7fe2a013f650 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: _finish_auth 0 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe290001690 con 0x7fe2a013f650 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a4dcb640 10 monclient: dump: 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:19.222 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a8031640 5 monclient: authenticate success, global_id 4242 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.238+0000 7fe2a8031640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2900017f0 con 0x7fe2a013f650 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a4dcb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a4dcb640 1 -- 172.21.15.45:0/2172014059 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe290001c40 con 0x7fe2a013f650 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a4dcb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a4dcb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:19.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a4dcb640 10 monclient: dump: 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a8031640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:19.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.239+0000 7fe2a8031640 4 set_mon_vals no callback set 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 10 monclient: shutdown 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 1 -- 172.21.15.45:0/2172014059 >> v1:172.21.15.45:6789/0 conn(0x7fe2a013f650 legacy=0x7fe2a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 1 -- 172.21.15.45:0/2172014059 shutdown_connections 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 1 -- 172.21.15.45:0/2172014059 >> 172.21.15.45:0/2172014059 conn(0x7fe2a0066a00 msgr2=0x7fe2a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 1 -- 172.21.15.45:0/2172014059 shutdown_connections 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.240+0000 7fe2a8031640 1 -- 172.21.15.45:0/2172014059 wait complete. 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 10 monclient: build_initial_monmap 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 10 monclient: monmap: 2024-03-29T10:52:19.231 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 1 librados: starting msgr at 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 1 librados: starting objecter 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 1 Processor -- start 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 1 -- start start 2024-03-29T10:52:19.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 1 librados: setting wanted keys 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 1 librados: calling monclient init 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.241+0000 7fe2a8031640 10 monclient: init 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient(hunting): picked mon.noname-a con 0x7fe2a00a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient(hunting): picked mon.noname-c con 0x7fe2a013f650 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient(hunting): picked mon.noname-b con 0x7fe2a01457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a00b1830 con 0x7fe2a00a9d70 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a0142a60 con 0x7fe2a01457f0 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a0142540 con 0x7fe2a013f650 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a8031640 10 monclient(hunting): authenticate will time out at 813.511766s 2024-03-29T10:52:19.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a55cc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe2a013f650 0x7fe2a01d7010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34548/0 (socket says 172.21.15.45:34548) 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2a55cc640 1 -- 172.21.15.45:0/1135497182 learned_addr learned my addr 172.21.15.45:0/1135497182 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.242+0000 7fe2967fc640 10 client.?.objecter ms_handle_connect 0x7fe2a013f650 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659859092 0 0) 0x7fe2a0142540 con 0x7fe2a013f650 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe284004eb0 con 0x7fe2a013f650 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 client.?.objecter ms_handle_connect 0x7fe2a01457f0 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 client.?.objecter ms_handle_connect 0x7fe2a00a9d70 2024-03-29T10:52:19.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925781940 0 0) 0x7fe2a0142a60 con 0x7fe2a01457f0 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a0142540 con 0x7fe2a01457f0 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017210879 0 0) 0x7fe284004eb0 con 0x7fe2a013f650 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2a0142a60 con 0x7fe2a013f650 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe290001c40 con 0x7fe2a013f650 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): dump: 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503965130 0 0) 0x7fe2a00b1830 con 0x7fe2a00a9d70 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.243+0000 7fe2967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe284004eb0 con 0x7fe2a00a9d70 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 841937222 0 0) 0x7fe2a0142540 con 0x7fe2a01457f0 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2a00b1830 con 0x7fe2a01457f0 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe298001a30 con 0x7fe2a01457f0 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient(hunting): dump: 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.237 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3774289732 0 0) 0x7fe2a00b1830 con 0x7fe2a01457f0 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 >> v1:172.21.15.45:6790/0 conn(0x7fe2a013f650 legacy=0x7fe2a01d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 >> v1:172.21.15.45:6789/0 conn(0x7fe2a00a9d70 legacy=0x7fe2a00b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 monclient: found mon.b 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00b1300 con 0x7fe2a01457f0 2024-03-29T10:52:19.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.244+0000 7fe2a8031640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 10 monclient: _renew_subs 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a00b14f0 con 0x7fe2a01457f0 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 10 monclient: _renew_subs 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a0141bb0 con 0x7fe2a01457f0 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.245+0000 7fe2a8031640 1 librados: init done 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.246+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe298002080 con 0x7fe2a01457f0 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.246+0000 7fe2967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.246+0000 7fe297fff640 4 set_mon_vals no callback set 2024-03-29T10:52:19.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.246+0000 7fe2a8031640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe298003b10 con 0x7fe2a01457f0 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2967fc640 10 monclient: dump: 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:19.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe298019b20 con 0x7fe2a01457f0 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2a8031640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2a8031640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.248+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe288005080 con 0x7fe2a01457f0 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.249+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fe298083b30 con 0x7fe2a01457f0 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.249+0000 7fe2967fc640 10 client.4240.objecter ms_dispatch 0x7fe2a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.249+0000 7fe2967fc640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.249+0000 7fe2967fc640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:19.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.249+0000 7fe2967fc640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T10:52:19.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.252+0000 7fe2a5dcd640 10 monclient: get_auth_request con 0x7fe28406dfc0 auth_method 0 2024-03-29T10:52:19.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.252+0000 7fe2967fc640 10 client.4240.objecter ms_handle_connect 0x7fe28406dfc0 2024-03-29T10:52:19.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.257+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe29804f2c0 con 0x7fe2a01457f0 2024-03-29T10:52:19.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.257+0000 7fe2967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.257+0000 7fe2967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:19.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.607+0000 7fe2a8031640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.607+0000 7fe2a8031640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.607+0000 7fe2a8031640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:19.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.607+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe288005330 con 0x7fe2a01457f0 2024-03-29T10:52:19.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.608+0000 7fe2967fc640 1 -- 172.21.15.45:0/1135497182 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fe2980513c0 con 0x7fe2a01457f0 2024-03-29T10:52:19.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.608+0000 7fe2967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.608+0000 7fe2967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:19.590 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:19.590 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:19.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 10 librados: watch_flush enter 2024-03-29T10:52:19.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 10 librados: watch_flush exit 2024-03-29T10:52:19.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T10:52:19.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 10 client.4240.objecter successfully canceled tick 2024-03-29T10:52:19.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe28406dfc0 legacy=0x7fe284070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 10 monclient: shutdown 2024-03-29T10:52:19.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:19.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 >> v1:172.21.15.116:6789/0 conn(0x7fe2a01457f0 legacy=0x7fe2a01e2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 shutdown_connections 2024-03-29T10:52:19.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 >> 172.21.15.45:0/1135497182 conn(0x7fe2a0066a00 msgr2=0x7fe2a00b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:19.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 shutdown_connections 2024-03-29T10:52:19.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 -- 172.21.15.45:0/1135497182 wait complete. 2024-03-29T10:52:19.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:19.611+0000 7fe2a8031640 1 librados: shutdown 2024-03-29T10:52:19.608 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:26.415 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.568+0000 7f6a381e8640 10 monclient: get_monmap_and_config 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.568+0000 7f6a381e8640 10 monclient: build_initial_monmap 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.568+0000 7f6a381e8640 10 monclient: monmap: 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:26.569683+0000 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:26.569683+0000 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:26.550 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.569+0000 7f6a381e8640 1 Processor -- start 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.570+0000 7f6a381e8640 1 -- start start 2024-03-29T10:52:26.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.570+0000 7f6a381e8640 10 monclient: init 2024-03-29T10:52:26.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:26.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:26.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 10 monclient(hunting): picked mon.noname-b con 0x7f6a301422b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a35f84640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6a301422b0 0x7f6a3013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56550/0 (socket says 172.21.15.45:56550) 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a35f84640 1 -- 172.21.15.45:0/3592717244 learned_addr learned my addr 172.21.15.45:0/3592717244 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 10 monclient(hunting): picked mon.noname-a con 0x7f6a30142690 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 10 monclient(hunting): picked mon.noname-c con 0x7f6a300b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a30142c80 con 0x7f6a30142690 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300aae40 con 0x7f6a300b17d0 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a3013fbd0 con 0x7f6a301422b0 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.571+0000 7f6a381e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.572+0000 7f6a381e8640 10 monclient(hunting): authenticate will time out at 820.840995s 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.572+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178944350 0 0) 0x7f6a3013fbd0 con 0x7f6a301422b0 2024-03-29T10:52:26.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a1c003520 con 0x7f6a301422b0 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643440063 0 0) 0x7f6a1c003520 con 0x7f6a301422b0 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 >> v1:172.21.15.116:6789/0 conn(0x7f6a300b17d0 legacy=0x7f6a300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.573+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 >> v1:172.21.15.45:6789/0 conn(0x7f6a30142690 legacy=0x7f6a300b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: _finish_hunting 0 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 1 monclient: found mon.noname-b 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30151940 con 0x7f6a301422b0 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: _finish_auth 0 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: _check_auth_tickets 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6a20001690 con 0x7f6a301422b0 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: dump: 2024-03-29T10:52:26.557 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a381e8640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a381e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a200017f0 con 0x7f6a301422b0 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 1 -- 172.21.15.45:0/3592717244 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6a20001c40 con 0x7f6a301422b0 2024-03-29T10:52:26.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a34f82640 10 monclient: dump: 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.575+0000 7f6a381e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:26.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.576+0000 7f6a381e8640 4 set_mon_vals no callback set 2024-03-29T10:52:26.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.576+0000 7f6a381e8640 10 monclient: shutdown 2024-03-29T10:52:26.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.576+0000 7f6a381e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:26.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.576+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 >> v1:172.21.15.45:6790/0 conn(0x7f6a301422b0 legacy=0x7f6a3013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.577+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 shutdown_connections 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.577+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 >> 172.21.15.45:0/3592717244 conn(0x7f6a30066a00 msgr2=0x7f6a30066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.577+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 shutdown_connections 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.577+0000 7f6a381e8640 1 -- 172.21.15.45:0/3592717244 wait complete. 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 10 monclient: build_initial_monmap 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 10 monclient: monmap: 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:26.562 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 1 librados: starting msgr at 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 1 librados: starting objecter 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 1 Processor -- start 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 1 -- start start 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 1 librados: setting wanted keys 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 1 librados: calling monclient init 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.578+0000 7f6a381e8640 10 monclient: init 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient(hunting): picked mon.noname-b con 0x7f6a300b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:26.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient(hunting): picked mon.noname-c con 0x7f6a301422b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:26.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient(hunting): picked mon.noname-a con 0x7f6a30142690 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:26.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300abca0 con 0x7f6a30142690 2024-03-29T10:52:26.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300abf10 con 0x7f6a300b17d0 2024-03-29T10:52:26.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300ac1f0 con 0x7f6a301422b0 2024-03-29T10:52:26.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:26.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.579+0000 7f6a381e8640 10 monclient(hunting): authenticate will time out at 820.848773s 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.580+0000 7f6a35783640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6a301422b0 0x7f6a30132200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60584/0 (socket says 172.21.15.45:60584) 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.580+0000 7f6a35783640 1 -- 172.21.15.45:0/851720509 learned_addr learned my addr 172.21.15.45:0/851720509 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 client.?.objecter ms_handle_connect 0x7f6a301422b0 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 client.?.objecter ms_handle_connect 0x7f6a300b17d0 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379276458 0 0) 0x7f6a300abf10 con 0x7f6a300b17d0 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a14004eb0 con 0x7f6a300b17d0 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457796125 0 0) 0x7f6a300ac1f0 con 0x7f6a301422b0 2024-03-29T10:52:26.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a300abf10 con 0x7f6a301422b0 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242028807 0 0) 0x7f6a300abf10 con 0x7f6a301422b0 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a300ac1f0 con 0x7f6a301422b0 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6a2c001710 con 0x7f6a301422b0 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): dump: 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.570 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2848132870 0 0) 0x7f6a14004eb0 con 0x7f6a300b17d0 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.581+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a300abf10 con 0x7f6a300b17d0 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6a20001b00 con 0x7f6a300b17d0 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 10 monclient(hunting): dump: 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:26.571 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3421779113 0 0) 0x7f6a300ac1f0 con 0x7f6a301422b0 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 >> v1:172.21.15.116:6789/0 conn(0x7f6a300b17d0 legacy=0x7f6a301d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.582+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 >> v1:172.21.15.45:6789/0 conn(0x7f6a30142690 legacy=0x7f6a300b0730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 1 monclient: found mon.c 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:26.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a301db040 con 0x7f6a301422b0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a381e8640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a381e8640 10 monclient: _renew_subs 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a381e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.583+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a301d6ef0 con 0x7f6a301422b0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a381e8640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a381e8640 10 monclient: _renew_subs 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a381e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a30132af0 con 0x7f6a301422b0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a381e8640 1 librados: init done 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a2c001c20 con 0x7f6a301422b0 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:26.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6a2c004a00 con 0x7f6a301422b0 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a267fc640 10 monclient: dump: 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.584+0000 7f6a27fff640 4 set_mon_vals no callback set 2024-03-29T10:52:26.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.585+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f6a2c01a9d0 con 0x7f6a301422b0 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.586+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f6a2c084950 con 0x7f6a301422b0 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.586+0000 7f6a267fc640 10 client.4250.objecter ms_dispatch 0x7f6a30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.586+0000 7f6a267fc640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.586+0000 7f6a267fc640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.586+0000 7f6a267fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.586+0000 7f6a35f84640 10 monclient: get_auth_request con 0x7f6a1406de70 auth_method 0 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.587+0000 7f6a381e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.587+0000 7f6a381e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.587+0000 7f6a381e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.587+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a301db1d0 con 0x7f6a301422b0 2024-03-29T10:52:26.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.587+0000 7f6a267fc640 10 client.4250.objecter ms_handle_connect 0x7f6a1406de70 2024-03-29T10:52:26.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.596+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a2c050290 con 0x7f6a301422b0 2024-03-29T10:52:26.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.596+0000 7f6a267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.596+0000 7f6a267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.909+0000 7f6a381e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.909+0000 7f6a381e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.909+0000 7f6a381e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.909+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a30075f10 con 0x7f6a301422b0 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.910+0000 7f6a267fc640 1 -- 172.21.15.45:0/851720509 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f6a2c084000 con 0x7f6a301422b0 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.910+0000 7f6a267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:26.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.910+0000 7f6a267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:26.892 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:26.892 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:26.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 10 librados: watch_flush enter 2024-03-29T10:52:26.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 10 librados: watch_flush exit 2024-03-29T10:52:26.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 10 client.4250.objecter successfully canceled tick 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 >> v1:172.21.15.116:6816/3290579615 conn(0x7f6a1406de70 legacy=0x7f6a14070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 10 monclient: shutdown 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 >> v1:172.21.15.45:6790/0 conn(0x7f6a301422b0 legacy=0x7f6a30132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 shutdown_connections 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.913+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 >> 172.21.15.45:0/851720509 conn(0x7f6a30066a00 msgr2=0x7f6a300b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.914+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 shutdown_connections 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.914+0000 7f6a381e8640 1 -- 172.21.15.45:0/851720509 wait complete. 2024-03-29T10:52:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:26.914+0000 7f6a381e8640 1 librados: shutdown 2024-03-29T10:52:26.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:33.716 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:33.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.887+0000 7f2288aed640 10 monclient: get_monmap_and_config 2024-03-29T10:52:33.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.887+0000 7f2288aed640 10 monclient: build_initial_monmap 2024-03-29T10:52:33.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.887+0000 7f2288aed640 10 monclient: monmap: 2024-03-29T10:52:33.869 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:33.869 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:33.869 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:33.888680+0000 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:33.888680+0000 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:33.870 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.892+0000 7f2288aed640 1 Processor -- start 2024-03-29T10:52:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.892+0000 7f2288aed640 1 -- start start 2024-03-29T10:52:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.892+0000 7f2288aed640 10 monclient: init 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient(hunting): picked mon.noname-a con 0x7f228406d9d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient(hunting): picked mon.noname-b con 0x7f228413ef30 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient(hunting): picked mon.noname-c con 0x7f22840b0860 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22840aed10 con 0x7f228406d9d0 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22840ab1b0 con 0x7f22840b0860 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22840ab490 con 0x7f228413ef30 2024-03-29T10:52:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.893+0000 7f2288aed640 10 monclient(hunting): authenticate will time out at 828.162442s 2024-03-29T10:52:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f2282d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f22840b0860 0x7f2284142a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:58661/0 (socket says 172.21.15.45:58661) 2024-03-29T10:52:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f2282d9d640 1 -- 172.21.15.45:0/353294322 learned_addr learned my addr 172.21.15.45:0/353294322 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485570732 0 0) 0x7f22840ab1b0 con 0x7f22840b0860 2024-03-29T10:52:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f228159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f228159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f228159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f228159a640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:52:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.896+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c003520 con 0x7f22840b0860 2024-03-29T10:52:33.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.897+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597948022 0 0) 0x7f226c003520 con 0x7f22840b0860 2024-03-29T10:52:33.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.897+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 >> v1:172.21.15.45:6790/0 conn(0x7f228413ef30 legacy=0x7f22840af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 >> v1:172.21.15.45:6789/0 conn(0x7f228406d9d0 legacy=0x7f228413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 1 monclient: found mon.noname-c 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840ab770 con 0x7f22840b0860 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: _finish_auth 0 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:33.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2274001690 con 0x7f22840b0860 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: dump: 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:33.884 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:33.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2274001860 con 0x7f22840b0860 2024-03-29T10:52:33.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:33.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 1 -- 172.21.15.45:0/353294322 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2274001cb0 con 0x7f22840b0860 2024-03-29T10:52:33.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:33.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.901+0000 7f228159a640 10 monclient: dump: 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:33.886 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.902+0000 7f2288aed640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.902+0000 7f2288aed640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.902+0000 7f2288aed640 4 set_mon_vals no callback set 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 10 monclient: shutdown 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 1 -- 172.21.15.45:0/353294322 >> v1:172.21.15.116:6789/0 conn(0x7f22840b0860 legacy=0x7f2284142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 1 -- 172.21.15.45:0/353294322 shutdown_connections 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 1 -- 172.21.15.45:0/353294322 >> 172.21.15.45:0/353294322 conn(0x7f228406b660 msgr2=0x7f228406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:33.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 1 -- 172.21.15.45:0/353294322 shutdown_connections 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.903+0000 7f2288aed640 1 -- 172.21.15.45:0/353294322 wait complete. 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.904+0000 7f2288aed640 10 monclient: build_initial_monmap 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.904+0000 7f2288aed640 10 monclient: monmap: 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:33.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 1 librados: starting msgr at 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 1 librados: starting objecter 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 1 Processor -- start 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 1 -- start start 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 1 librados: setting wanted keys 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 1 librados: calling monclient init 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.905+0000 7f2288aed640 10 monclient: init 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient(hunting): picked mon.noname-b con 0x7f228406d9d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient(hunting): picked mon.noname-a con 0x7f22840b0860 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:33.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient(hunting): picked mon.noname-c con 0x7f228413ef30 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:33.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228406fc70 con 0x7f22840b0860 2024-03-29T10:52:33.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22841dbd20 con 0x7f228406d9d0 2024-03-29T10:52:33.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228406ff50 con 0x7f228413ef30 2024-03-29T10:52:33.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:33.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.906+0000 7f2288aed640 10 monclient(hunting): authenticate will time out at 828.175515s 2024-03-29T10:52:33.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.911+0000 7f2281d9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f22840b0860 0x7f22841daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38485/0 (socket says 172.21.15.45:38485) 2024-03-29T10:52:33.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.911+0000 7f228259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f228406d9d0 0x7f228414dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:58735/0 (socket says 172.21.15.45:58735) 2024-03-29T10:52:33.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.911+0000 7f228259c640 1 -- 172.21.15.45:0/3206784017 learned_addr learned my addr 172.21.15.45:0/3206784017 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:33.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 10 client.?.objecter ms_handle_connect 0x7f228406d9d0 2024-03-29T10:52:33.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:33.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840324229 0 0) 0x7f22841dbd20 con 0x7f228406d9d0 2024-03-29T10:52:33.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:33.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:33.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:33.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:52:33.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.912+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2264004eb0 con 0x7f228406d9d0 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.914+0000 7f2272ffd640 10 client.?.objecter ms_handle_connect 0x7f22840b0860 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.914+0000 7f2272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.914+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223667334 0 0) 0x7f2264004eb0 con 0x7f228406d9d0 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22841dbd20 con 0x7f228406d9d0 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f227c001d50 con 0x7f228406d9d0 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:33.896 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:33.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:33.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2145972312 0 0) 0x7f22841dbd20 con 0x7f228406d9d0 2024-03-29T10:52:33.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 >> v1:172.21.15.45:6790/0 conn(0x7f228413ef30 legacy=0x7f22841db610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:33.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 >> v1:172.21.15.45:6789/0 conn(0x7f22840b0860 legacy=0x7f22841daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:33.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 1 monclient: found mon.b 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.915+0000 7f2272ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228414e3c0 con 0x7f228406d9d0 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 10 client.?.objecter ms_handle_connect 0x7f228413ef30 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2272ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 10 monclient: _renew_subs 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2284150680 con 0x7f228406d9d0 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 10 monclient: _renew_subs 2024-03-29T10:52:33.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2284071a00 con 0x7f228406d9d0 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.916+0000 7f2288aed640 1 librados: init done 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2288aed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2288aed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2288aed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2284075f10 con 0x7f228406d9d0 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f227c002010 con 0x7f228406d9d0 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2272ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f227c003de0 con 0x7f228406d9d0 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.918+0000 7f2280d99640 4 set_mon_vals no callback set 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.919+0000 7f2272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.919+0000 7f2272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.919+0000 7f2272ffd640 10 monclient: dump: 2024-03-29T10:52:33.902 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:33.903 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:33.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.924+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f227c003f40 con 0x7f228406d9d0 2024-03-29T10:52:33.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.925+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f227c084df0 con 0x7f228406d9d0 2024-03-29T10:52:33.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.925+0000 7f2272ffd640 10 client.4252.objecter ms_dispatch 0x7f2284000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:33.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.925+0000 7f2281d9b640 10 monclient: get_auth_request con 0x7f226406dd00 auth_method 0 2024-03-29T10:52:33.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.925+0000 7f2272ffd640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:33.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.925+0000 7f2272ffd640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:33.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.925+0000 7f2272ffd640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T10:52:33.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.929+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f227c0506d0 con 0x7f228406d9d0 2024-03-29T10:52:33.911 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.930+0000 7f2272ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:33.911 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.930+0000 7f2272ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:33.911 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:33.930+0000 7f2272ffd640 10 client.4252.objecter ms_handle_connect 0x7f226406dd00 2024-03-29T10:52:34.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.262+0000 7f2288aed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.262+0000 7f2288aed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.262+0000 7f2288aed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:52:34.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.262+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2284072330 con 0x7f228406d9d0 2024-03-29T10:52:34.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.263+0000 7f2272ffd640 1 -- 172.21.15.45:0/3206784017 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f227c052600 con 0x7f228406d9d0 2024-03-29T10:52:34.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.263+0000 7f2272ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.263+0000 7f2272ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:34.245 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:34.245 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:34.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 10 librados: watch_flush enter 2024-03-29T10:52:34.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 10 librados: watch_flush exit 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 10 client.4252.objecter successfully canceled tick 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 >> v1:172.21.15.116:6816/3290579615 conn(0x7f226406dd00 legacy=0x7f22640701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 10 monclient: shutdown 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.267+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 >> v1:172.21.15.116:6789/0 conn(0x7f228406d9d0 legacy=0x7f228414dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.269+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 shutdown_connections 2024-03-29T10:52:34.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.269+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 >> 172.21.15.45:0/3206784017 conn(0x7f228406b660 msgr2=0x7f22840ac9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:34.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.269+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 shutdown_connections 2024-03-29T10:52:34.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.269+0000 7f2288aed640 1 -- 172.21.15.45:0/3206784017 wait complete. 2024-03-29T10:52:34.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:34.269+0000 7f2288aed640 1 librados: shutdown 2024-03-29T10:52:34.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:41.076 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:41.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.241+0000 7fd273fff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.241+0000 7fd273fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.241+0000 7fd273fff640 10 monclient: monmap: 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:41.242089+0000 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:41.242089+0000 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:41.223 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:41.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.243+0000 7fd273fff640 1 Processor -- start 2024-03-29T10:52:41.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.243+0000 7fd273fff640 1 -- start start 2024-03-29T10:52:41.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.243+0000 7fd273fff640 10 monclient: init 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.244+0000 7fd273fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.244+0000 7fd273fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.244+0000 7fd273fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd2740a9750 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.244+0000 7fd273fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd2740add40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.246+0000 7fd2737fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd2740aad20 0x7fd274151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:55209/0 (socket says 172.21.15.45:55209) 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.246+0000 7fd2737fe640 1 -- 172.21.15.45:0/1335998117 learned_addr learned my addr 172.21.15.45:0/1335998117 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd273fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd2740aad20 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd274149870 con 0x7fd2740a9750 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2740abf80 con 0x7fd2740aad20 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd274149ac0 con 0x7fd2740add40 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd273fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd273fff640 10 monclient(hunting): authenticate will time out at 835.516296s 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516426947 0 0) 0x7fd2740abf80 con 0x7fd2740aad20 2024-03-29T10:52:41.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:52:41.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c003520 con 0x7fd2740aad20 2024-03-29T10:52:41.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108049505 0 0) 0x7fd25c003520 con 0x7fd2740aad20 2024-03-29T10:52:41.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.247+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 >> v1:172.21.15.45:6790/0 conn(0x7fd2740add40 legacy=0x7fd274143610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 >> v1:172.21.15.45:6789/0 conn(0x7fd2740a9750 legacy=0x7fd274149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 1 monclient: found mon.noname-c 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27414a6f0 con 0x7fd2740aad20 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:41.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd26c001a10 con 0x7fd2740aad20 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: dump: 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:41.232 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd273fff640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd273fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd26c001ca0 con 0x7fd2740aad20 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 1 -- 172.21.15.45:0/1335998117 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd26c002030 con 0x7fd2740aad20 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd271ffb640 10 monclient: dump: 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.233 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd273fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.248+0000 7fd273fff640 4 set_mon_vals no callback set 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.249+0000 7fd273fff640 10 monclient: shutdown 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.249+0000 7fd273fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.249+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 >> v1:172.21.15.116:6789/0 conn(0x7fd2740aad20 legacy=0x7fd274151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.249+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 shutdown_connections 2024-03-29T10:52:41.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.249+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 >> 172.21.15.45:0/1335998117 conn(0x7fd2740673e0 msgr2=0x7fd2740b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:41.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.250+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 shutdown_connections 2024-03-29T10:52:41.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.254+0000 7fd273fff640 1 -- 172.21.15.45:0/1335998117 wait complete. 2024-03-29T10:52:41.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 10 monclient: monmap: 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 1 librados: starting msgr at 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 1 librados: starting objecter 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 1 Processor -- start 2024-03-29T10:52:41.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 1 -- start start 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 1 librados: setting wanted keys 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 1 librados: calling monclient init 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.255+0000 7fd273fff640 10 monclient: init 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.256+0000 7fd273fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.256+0000 7fd273fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.256+0000 7fd273fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd2740a9750 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.256+0000 7fd273fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd2740aad20 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.257+0000 7fd273fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd2740add40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.257+0000 7fd273fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27413ef00 con 0x7fd2740a9750 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.257+0000 7fd273fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd274139de0 con 0x7fd2740aad20 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.257+0000 7fd273fff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27408bbd0 con 0x7fd2740add40 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.257+0000 7fd273fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:41.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.257+0000 7fd273fff640 10 monclient(hunting): authenticate will time out at 835.526451s 2024-03-29T10:52:41.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd2737fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd2740add40 0x7fd2741f76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39903/0 (socket says 172.21.15.45:39903) 2024-03-29T10:52:41.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd2737fe640 1 -- 172.21.15.45:0/3967035382 learned_addr learned my addr 172.21.15.45:0/3967035382 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:41.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 client.?.objecter ms_handle_connect 0x7fd2740add40 2024-03-29T10:52:41.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:41.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 client.?.objecter ms_handle_connect 0x7fd2740a9750 2024-03-29T10:52:41.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209709932 0 0) 0x7fd27408bbd0 con 0x7fd2740add40 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd260004eb0 con 0x7fd2740add40 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217085833 0 0) 0x7fd260004eb0 con 0x7fd2740add40 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd27408bbd0 con 0x7fd2740add40 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd26c001990 con 0x7fd2740add40 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:41.243 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:41.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.261+0000 7fd25bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060475922 0 0) 0x7fd27413ef00 con 0x7fd2740a9750 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd260004eb0 con 0x7fd2740a9750 2024-03-29T10:52:41.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1279096855 0 0) 0x7fd27408bbd0 con 0x7fd2740add40 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.262+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 >> v1:172.21.15.116:6789/0 conn(0x7fd2740aad20 legacy=0x7fd27413a300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 >> v1:172.21.15.45:6789/0 conn(0x7fd2740a9750 legacy=0x7fd27413e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 1 monclient: found mon.c 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27406d8c0 con 0x7fd2740add40 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd25bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd273fff640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd273fff640 10 monclient: _renew_subs 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd273fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.263+0000 7fd273fff640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27406de20 con 0x7fd2740add40 2024-03-29T10:52:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.264+0000 7fd273fff640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:41.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.264+0000 7fd273fff640 10 monclient: _renew_subs 2024-03-29T10:52:41.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.264+0000 7fd273fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:41.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.264+0000 7fd273fff640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd274072680 con 0x7fd2740add40 2024-03-29T10:52:41.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.264+0000 7fd273fff640 1 librados: init done 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd259ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd259ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd259ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd254005080 con 0x7fd2740add40 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd26c004f70 con 0x7fd2740add40 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd26c005280 con 0x7fd2740add40 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 10 monclient: dump: 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fd26c01b250 con 0x7fd2740add40 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.266+0000 7fd2717fa640 4 set_mon_vals no callback set 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.267+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fd26c085130 con 0x7fd2740add40 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.267+0000 7fd25bfff640 10 client.4259.objecter ms_dispatch 0x7fd274000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.267+0000 7fd25bfff640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.267+0000 7fd25bfff640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.267+0000 7fd25bfff640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.268+0000 7fd272ffd640 10 monclient: get_auth_request con 0x7fd26006de10 auth_method 0 2024-03-29T10:52:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.268+0000 7fd25bfff640 10 client.4259.objecter ms_handle_connect 0x7fd26006de10 2024-03-29T10:52:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.275+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd26c050a90 con 0x7fd2740add40 2024-03-29T10:52:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.275+0000 7fd25bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.275+0000 7fd25bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:41.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.598+0000 7fd259ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.598+0000 7fd259ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.598+0000 7fd259ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:52:41.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.598+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd254005330 con 0x7fd2740add40 2024-03-29T10:52:41.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.600+0000 7fd25bfff640 1 -- 172.21.15.45:0/3967035382 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fd26c0529c0 con 0x7fd2740add40 2024-03-29T10:52:41.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.600+0000 7fd25bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.600+0000 7fd25bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:41.586 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:41.586 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.603+0000 7fd259ffb640 10 librados: watch_flush enter 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 10 librados: watch_flush exit 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 10 client.4259.objecter successfully canceled tick 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 >> v1:172.21.15.116:6816/3290579615 conn(0x7fd26006de10 legacy=0x7fd2600702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 10 monclient: shutdown 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.604+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 >> v1:172.21.15.45:6790/0 conn(0x7fd2740add40 legacy=0x7fd2741f76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.605+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 shutdown_connections 2024-03-29T10:52:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.605+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 >> 172.21.15.45:0/3967035382 conn(0x7fd2740673e0 msgr2=0x7fd274069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.605+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 shutdown_connections 2024-03-29T10:52:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.606+0000 7fd259ffb640 1 -- 172.21.15.45:0/3967035382 wait complete. 2024-03-29T10:52:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:41.606+0000 7fd259ffb640 1 librados: shutdown 2024-03-29T10:52:41.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:48.412 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.580+0000 7fc9c759e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.580+0000 7fc9c759e640 10 monclient: build_initial_monmap 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.580+0000 7fc9c759e640 10 monclient: monmap: 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:48.581422+0000 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:48.581422+0000 2024-03-29T10:52:48.566 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:48.567 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:48.567 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:48.567 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:48.567 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:48.567 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.585+0000 7fc9c759e640 1 Processor -- start 2024-03-29T10:52:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.585+0000 7fc9c759e640 1 -- start start 2024-03-29T10:52:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.585+0000 7fc9c759e640 10 monclient: init 2024-03-29T10:52:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.586+0000 7fc9c759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.586+0000 7fc9c759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.586+0000 7fc9c759e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9c8071f40 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.586+0000 7fc9c759e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9c80adcb0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c759e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9c813f910 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c80af000 con 0x7fc9c813f910 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c759e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c80ab060 con 0x7fc9c80adcb0 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c759e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c80aec40 con 0x7fc9c8071f40 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c759e640 10 monclient(hunting): authenticate will time out at 842.860315s 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc9c8071f40 0x7fc9c813f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35761/0 (socket says 172.21.15.45:35761) 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c659c640 1 -- 172.21.15.45:0/2677178060 learned_addr learned my addr 172.21.15.45:0/2677178060 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129485176 0 0) 0x7fc9c80aec40 con 0x7fc9c8071f40 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c559a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c559a640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:52:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.591+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b0003520 con 0x7fc9c8071f40 2024-03-29T10:52:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808119033 0 0) 0x7fc9b0003520 con 0x7fc9c8071f40 2024-03-29T10:52:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 >> v1:172.21.15.116:6789/0 conn(0x7fc9c80adcb0 legacy=0x7fc9c8145560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 >> v1:172.21.15.45:6789/0 conn(0x7fc9c813f910 legacy=0x7fc9c8145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 1 monclient: found mon.noname-b 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c80b0030 con 0x7fc9c8071f40 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: _finish_auth 0 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc9b8001690 con 0x7fc9c8071f40 2024-03-29T10:52:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c559a640 10 monclient: dump: 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c759e640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.592+0000 7fc9c759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9b8001860 con 0x7fc9c8071f40 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c559a640 1 -- 172.21.15.45:0/2677178060 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc9b8001cb0 con 0x7fc9c8071f40 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c559a640 10 monclient: dump: 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:48.577 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.593+0000 7fc9c759e640 4 set_mon_vals no callback set 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.594+0000 7fc9c759e640 10 monclient: shutdown 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.594+0000 7fc9c759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:48.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.594+0000 7fc9c759e640 1 -- 172.21.15.45:0/2677178060 >> v1:172.21.15.45:6790/0 conn(0x7fc9c8071f40 legacy=0x7fc9c813f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.598+0000 7fc9c759e640 1 -- 172.21.15.45:0/2677178060 shutdown_connections 2024-03-29T10:52:48.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.598+0000 7fc9c759e640 1 -- 172.21.15.45:0/2677178060 >> 172.21.15.45:0/2677178060 conn(0x7fc9c80673e0 msgr2=0x7fc9c80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:48.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.604+0000 7fc9c759e640 1 -- 172.21.15.45:0/2677178060 shutdown_connections 2024-03-29T10:52:48.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.604+0000 7fc9c759e640 1 -- 172.21.15.45:0/2677178060 wait complete. 2024-03-29T10:52:48.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.606+0000 7fc9c759e640 10 monclient: build_initial_monmap 2024-03-29T10:52:48.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.606+0000 7fc9c759e640 10 monclient: monmap: 2024-03-29T10:52:48.588 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:48.588 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:48.588 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:48.588 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.606+0000 7fc9c759e640 1 librados: starting msgr at 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.606+0000 7fc9c759e640 1 librados: starting objecter 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.607+0000 7fc9c759e640 1 Processor -- start 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.607+0000 7fc9c759e640 1 -- start start 2024-03-29T10:52:48.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.607+0000 7fc9c759e640 1 librados: setting wanted keys 2024-03-29T10:52:48.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.607+0000 7fc9c759e640 1 librados: calling monclient init 2024-03-29T10:52:48.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.607+0000 7fc9c759e640 10 monclient: init 2024-03-29T10:52:48.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.608+0000 7fc9c759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:48.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.608+0000 7fc9c759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:48.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.608+0000 7fc9c759e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9c8071f40 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:48.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.611+0000 7fc9c759e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9c80adcb0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:48.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.615+0000 7fc9c659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc9c8071f40 0x7fc9c8143320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41237/0 (socket says 172.21.15.45:41237) 2024-03-29T10:52:48.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c759e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9c813f910 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c759e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c812b950 con 0x7fc9c8071f40 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c759e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c813e740 con 0x7fc9c813f910 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c759e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c8135350 con 0x7fc9c80adcb0 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c759e640 10 monclient(hunting): authenticate will time out at 842.885511s 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c5d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc9c80adcb0 0x7fc9c81dec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35915/0 (socket says 172.21.15.45:35915) 2024-03-29T10:52:48.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.616+0000 7fc9c5d9b640 1 -- 172.21.15.45:0/570573886 learned_addr learned my addr 172.21.15.45:0/570573886 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:48.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.618+0000 7fc9b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc9c80adcb0 2024-03-29T10:52:48.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.618+0000 7fc9b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc9c8071f40 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791500710 0 0) 0x7fc9c812b950 con 0x7fc9c8071f40 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a8004eb0 con 0x7fc9c8071f40 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2021708935 0 0) 0x7fc9a8004eb0 con 0x7fc9c8071f40 2024-03-29T10:52:48.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9c812b950 con 0x7fc9c8071f40 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc9b8001b70 con 0x7fc9c8071f40 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.602 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:48.603 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:48.603 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:48.603 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:48.603 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:48.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.619+0000 7fc9b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1636247391 0 0) 0x7fc9c812b950 con 0x7fc9c8071f40 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 >> v1:172.21.15.45:6790/0 conn(0x7fc9c80adcb0 legacy=0x7fc9c81dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 >> v1:172.21.15.116:6789/0 conn(0x7fc9c813f910 legacy=0x7fc9c81e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 1 monclient: found mon.a 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c812bc50 con 0x7fc9c8071f40 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:48.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 10 monclient: _renew_subs 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c813eb60 con 0x7fc9c8071f40 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 10 monclient: _renew_subs 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c812b6e0 con 0x7fc9c8071f40 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.620+0000 7fc9c759e640 1 librados: init done 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.621+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9b8004bf0 con 0x7fc9c8071f40 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.621+0000 7fc9b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.621+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc9b8001b70 con 0x7fc9c8071f40 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.621+0000 7fc9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.621+0000 7fc9b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.621+0000 7fc9b6ffd640 10 monclient: dump: 2024-03-29T10:52:48.605 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:48.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.622+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fc9b801abd0 con 0x7fc9c8071f40 2024-03-29T10:52:48.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.623+0000 7fc9c5d9b640 10 monclient: get_auth_request con 0x7fc9a806e010 auth_method 0 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.623+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fc9b8054520 con 0x7fc9c8071f40 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.623+0000 7fc9b6ffd640 10 client.4257.objecter ms_dispatch 0x7fc9c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.623+0000 7fc9b6ffd640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.623+0000 7fc9b6ffd640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.623+0000 7fc9b6ffd640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.624+0000 7fc9b7fff640 4 set_mon_vals no callback set 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.625+0000 7fc9b6ffd640 10 client.4257.objecter ms_handle_connect 0x7fc9a806e010 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.625+0000 7fc9c759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.625+0000 7fc9c759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.625+0000 7fc9c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:48.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.625+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac005080 con 0x7fc9c8071f40 2024-03-29T10:52:48.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.635+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9b80507c0 con 0x7fc9c8071f40 2024-03-29T10:52:48.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.635+0000 7fc9b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.635+0000 7fc9b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:48.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.967+0000 7fc9c759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:48.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.967+0000 7fc9c759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:48.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.967+0000 7fc9c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:48.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.967+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9ac005330 con 0x7fc9c8071f40 2024-03-29T10:52:48.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.968+0000 7fc9b6ffd640 1 -- 172.21.15.45:0/570573886 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fc9b80507c0 con 0x7fc9c8071f40 2024-03-29T10:52:48.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.968+0000 7fc9b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:48.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.968+0000 7fc9b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:48.952 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:48.953 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:48.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 10 librados: watch_flush enter 2024-03-29T10:52:48.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 10 librados: watch_flush exit 2024-03-29T10:52:48.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T10:52:48.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 10 client.4257.objecter successfully canceled tick 2024-03-29T10:52:48.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 >> v1:172.21.15.116:6816/3290579615 conn(0x7fc9a806e010 legacy=0x7fc9a80704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 10 monclient: shutdown 2024-03-29T10:52:48.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:48.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.973+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 >> v1:172.21.15.45:6789/0 conn(0x7fc9c8071f40 legacy=0x7fc9c8143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.976+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 shutdown_connections 2024-03-29T10:52:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.976+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 >> 172.21.15.45:0/570573886 conn(0x7fc9c80673e0 msgr2=0x7fc9c8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.976+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 shutdown_connections 2024-03-29T10:52:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.976+0000 7fc9c759e640 1 -- 172.21.15.45:0/570573886 wait complete. 2024-03-29T10:52:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:48.976+0000 7fc9c759e640 1 librados: shutdown 2024-03-29T10:52:48.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:55.782 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.951+0000 7fb0f3fff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.951+0000 7fb0f3fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.951+0000 7fb0f3fff640 10 monclient: monmap: 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:52:55.952772+0000 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:52:55.952772+0000 2024-03-29T10:52:55.933 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:55.934 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:55.934 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:55.934 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:52:55.934 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:52:55.934 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.956+0000 7fb0f3fff640 1 Processor -- start 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.956+0000 7fb0f3fff640 1 -- start start 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.956+0000 7fb0f3fff640 10 monclient: init 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ec006eb0 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ec038f10 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ec036690 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec033c90 con 0x7fb0ec036690 2024-03-29T10:52:55.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0389f0 con 0x7fb0ec006eb0 2024-03-29T10:52:55.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec030890 con 0x7fb0ec038f10 2024-03-29T10:52:55.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:55.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.957+0000 7fb0f3fff640 10 monclient(hunting): authenticate will time out at 850.226243s 2024-03-29T10:52:55.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.961+0000 7fb0f27fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb0ec038f10 0x7fb0ec039330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55317/0 (socket says 172.21.15.45:55317) 2024-03-29T10:52:55.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.961+0000 7fb0f27fc640 1 -- 172.21.15.45:0/2831244791 learned_addr learned my addr 172.21.15.45:0/2831244791 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111984509 0 0) 0x7fb0ec030890 con 0x7fb0ec038f10 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0dc003520 con 0x7fb0ec038f10 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931055890 0 0) 0x7fb0ec033c90 con 0x7fb0ec036690 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:55.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:52:55.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec030890 con 0x7fb0ec036690 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437608834 0 0) 0x7fb0dc003520 con 0x7fb0ec038f10 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 >> v1:172.21.15.116:6789/0 conn(0x7fb0ec006eb0 legacy=0x7fb0ec031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 >> v1:172.21.15.45:6789/0 conn(0x7fb0ec036690 legacy=0x7fb0ec036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 monclient: found mon.noname-b 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:55.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0cbc40 con 0x7fb0ec038f10 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e4001690 con 0x7fb0ec038f10 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.962+0000 7fb0f1ffb640 10 monclient: dump: 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:55.947 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4001860 con 0x7fb0ec038f10 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f1ffb640 1 -- 172.21.15.45:0/2831244791 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e4001cb0 con 0x7fb0ec038f10 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f1ffb640 10 monclient: dump: 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.948 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f3fff640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f3fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:55.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.963+0000 7fb0f3fff640 4 set_mon_vals no callback set 2024-03-29T10:52:55.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.968+0000 7fb0f3fff640 10 monclient: shutdown 2024-03-29T10:52:55.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.968+0000 7fb0f3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:55.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.968+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2831244791 >> v1:172.21.15.45:6790/0 conn(0x7fb0ec038f10 legacy=0x7fb0ec039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.969+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2831244791 shutdown_connections 2024-03-29T10:52:55.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.969+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2831244791 >> 172.21.15.45:0/2831244791 conn(0x7fb0ec005c20 msgr2=0x7fb0ec006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:55.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.969+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2831244791 shutdown_connections 2024-03-29T10:52:55.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.970+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2831244791 wait complete. 2024-03-29T10:52:55.952 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.971+0000 7fb0f3fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:55.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.971+0000 7fb0f3fff640 10 monclient: monmap: 2024-03-29T10:52:55.953 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:52:55.953 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:55.953 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:52:55.953 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 1 librados: starting msgr at 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 1 librados: starting objecter 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 1 Processor -- start 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 1 -- start start 2024-03-29T10:52:55.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 1 librados: setting wanted keys 2024-03-29T10:52:55.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 1 librados: calling monclient init 2024-03-29T10:52:55.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.972+0000 7fb0f3fff640 10 monclient: init 2024-03-29T10:52:55.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:55.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:55.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ec006eb0 addr v1:172.21.15.45:6790/0 2024-03-29T10:52:55.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ec036690 addr v1:172.21.15.45:6789/0 2024-03-29T10:52:55.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ec038f10 addr v1:172.21.15.116:6789/0 2024-03-29T10:52:55.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec02da90 con 0x7fb0ec036690 2024-03-29T10:52:55.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0ceda0 con 0x7fb0ec038f10 2024-03-29T10:52:55.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0cf010 con 0x7fb0ec006eb0 2024-03-29T10:52:55.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:55.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.973+0000 7fb0f3fff640 10 monclient(hunting): authenticate will time out at 850.242167s 2024-03-29T10:52:55.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0f27fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb0ec036690 0x7fb0ec02e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36275/0 (socket says 172.21.15.45:36275) 2024-03-29T10:52:55.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0f27fc640 1 -- 172.21.15.45:0/2635665221 learned_addr learned my addr 172.21.15.45:0/2635665221 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:52:55.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 client.?.objecter ms_handle_connect 0x7fb0ec036690 2024-03-29T10:52:55.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571413598 0 0) 0x7fb0ec02da90 con 0x7fb0ec036690 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0004eb0 con 0x7fb0ec036690 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 client.?.objecter ms_handle_connect 0x7fb0ec006eb0 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.978+0000 7fb0db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2743396731 0 0) 0x7fb0e0004eb0 con 0x7fb0ec036690 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0ec02da90 con 0x7fb0ec036690 2024-03-29T10:52:55.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e4001b70 con 0x7fb0ec036690 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:55.967 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417982463 0 0) 0x7fb0ec0cf010 con 0x7fb0ec006eb0 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.979+0000 7fb0db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:52:55.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0004eb0 con 0x7fb0ec006eb0 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3615825571 0 0) 0x7fb0ec02da90 con 0x7fb0ec036690 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 >> v1:172.21.15.45:6790/0 conn(0x7fb0ec006eb0 legacy=0x7fb0ec02df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 >> v1:172.21.15.116:6789/0 conn(0x7fb0ec038f10 legacy=0x7fb0ec16bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 1 monclient: found mon.a 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0cf2f0 con 0x7fb0ec036690 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:55.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 10 monclient: _renew_subs 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec0cdf90 con 0x7fb0ec036690 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 10 monclient: _renew_subs 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec0cfa80 con 0x7fb0ec036690 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.980+0000 7fb0f3fff640 1 librados: init done 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 10 client.4263.objecter ms_handle_connect 0x7fb0ec038f10 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 10 client.4263.objecter resend_mon_ops 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4004a90 con 0x7fb0ec036690 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:55.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e4004e60 con 0x7fb0ec036690 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.981+0000 7fb0db7fe640 10 monclient: dump: 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.982+0000 7fb0f3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:55.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.982+0000 7fb0f3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.982+0000 7fb0f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.982+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec007b90 con 0x7fb0ec036690 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.983+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fb0e4004fc0 con 0x7fb0ec036690 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.984+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fb0e4084ae0 con 0x7fb0ec036690 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.984+0000 7fb0db7fe640 10 client.4263.objecter ms_dispatch 0x7fb0f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.984+0000 7fb0db7fe640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.984+0000 7fb0db7fe640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.984+0000 7fb0db7fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T10:52:55.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.984+0000 7fb0f17fa640 4 set_mon_vals no callback set 2024-03-29T10:52:55.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.985+0000 7fb0f2ffd640 10 monclient: get_auth_request con 0x7fb0e006dd90 auth_method 0 2024-03-29T10:52:55.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.986+0000 7fb0db7fe640 10 client.4263.objecter ms_handle_connect 0x7fb0e006dd90 2024-03-29T10:52:55.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.993+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0e40503c0 con 0x7fb0ec036690 2024-03-29T10:52:55.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.993+0000 7fb0db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:55.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:55.993+0000 7fb0db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:56.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.331+0000 7fb0f3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.332+0000 7fb0f3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.332+0000 7fb0f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:52:56.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.332+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0ec02dc70 con 0x7fb0ec036690 2024-03-29T10:52:56.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.333+0000 7fb0db7fe640 1 -- 172.21.15.45:0/2635665221 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fb0ec02dc70 con 0x7fb0ec036690 2024-03-29T10:52:56.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.333+0000 7fb0db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.333+0000 7fb0db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:56.315 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:52:56.315 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.336+0000 7fb0f3fff640 10 librados: watch_flush enter 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.336+0000 7fb0f3fff640 10 librados: watch_flush exit 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 10 client.4263.objecter successfully canceled tick 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 >> v1:172.21.15.116:6816/3290579615 conn(0x7fb0e006dd90 legacy=0x7fb0e0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 10 monclient: shutdown 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 >> v1:172.21.15.45:6789/0 conn(0x7fb0ec036690 legacy=0x7fb0ec02e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 shutdown_connections 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 >> 172.21.15.45:0/2635665221 conn(0x7fb0ec005c20 msgr2=0x7fb0ec0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.337+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 shutdown_connections 2024-03-29T10:52:56.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.338+0000 7fb0f3fff640 1 -- 172.21.15.45:0/2635665221 wait complete. 2024-03-29T10:52:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:52:56.338+0000 7fb0f3fff640 1 librados: shutdown 2024-03-29T10:52:56.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:03.144 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:03.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.317+0000 7f89ff57a640 10 monclient: get_monmap_and_config 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.317+0000 7f89ff57a640 10 monclient: build_initial_monmap 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.317+0000 7f89ff57a640 10 monclient: monmap: 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:03.318552+0000 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:03.318552+0000 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:03.299 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.318+0000 7f89ff57a640 1 Processor -- start 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.318+0000 7f89ff57a640 1 -- start start 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.318+0000 7f89ff57a640 10 monclient: init 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient(hunting): picked mon.noname-b con 0x7f89f81422b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient(hunting): picked mon.noname-a con 0x7f89f8142690 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient(hunting): picked mon.noname-c con 0x7f89f80b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f8142c80 con 0x7f89f8142690 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80aae40 con 0x7f89f80b17d0 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f813fbd0 con 0x7f89f81422b0 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:03.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.319+0000 7f89ff57a640 10 monclient(hunting): authenticate will time out at 857.588329s 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.321+0000 7f89fdb17640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f89f80b17d0 0x7f89f80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:41337/0 (socket says 172.21.15.45:41337) 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89fdb17640 1 -- 172.21.15.45:0/986048573 learned_addr learned my addr 172.21.15.45:0/986048573 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916828599 0 0) 0x7f89f80aae40 con 0x7f89f80b17d0 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e4003520 con 0x7f89f80b17d0 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153715990 0 0) 0x7f89f8142c80 con 0x7f89f8142690 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.322+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f80aae40 con 0x7f89f8142690 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 344813859 0 0) 0x7f89e4003520 con 0x7f89f80b17d0 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 >> v1:172.21.15.45:6790/0 conn(0x7f89f81422b0 legacy=0x7f89f813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 >> v1:172.21.15.45:6789/0 conn(0x7f89f8142690 legacy=0x7f89f80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 1 monclient: found mon.noname-c 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f8151940 con 0x7f89f80b17d0 2024-03-29T10:53:03.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: _finish_auth 0 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89f0001990 con 0x7f89f80b17d0 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89effff640 10 monclient: dump: 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:03.308 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:03.309 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:03.309 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:03.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89ff57a640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:53:03.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.323+0000 7f89ff57a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:03.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89f0001af0 con 0x7f89f80b17d0 2024-03-29T10:53:03.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:03.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89effff640 1 -- 172.21.15.45:0/986048573 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89f0001f40 con 0x7f89f80b17d0 2024-03-29T10:53:03.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89effff640 10 monclient: dump: 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.313 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89ff57a640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.325+0000 7f89ff57a640 4 set_mon_vals no callback set 2024-03-29T10:53:03.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.326+0000 7f89ff57a640 10 monclient: shutdown 2024-03-29T10:53:03.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.326+0000 7f89ff57a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:03.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.326+0000 7f89ff57a640 1 -- 172.21.15.45:0/986048573 >> v1:172.21.15.116:6789/0 conn(0x7f89f80b17d0 legacy=0x7f89f80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.327+0000 7f89ff57a640 1 -- 172.21.15.45:0/986048573 shutdown_connections 2024-03-29T10:53:03.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.327+0000 7f89ff57a640 1 -- 172.21.15.45:0/986048573 >> 172.21.15.45:0/986048573 conn(0x7f89f8066a00 msgr2=0x7f89f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:03.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.327+0000 7f89ff57a640 1 -- 172.21.15.45:0/986048573 shutdown_connections 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.327+0000 7f89ff57a640 1 -- 172.21.15.45:0/986048573 wait complete. 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.329+0000 7f89ff57a640 10 monclient: build_initial_monmap 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.329+0000 7f89ff57a640 10 monclient: monmap: 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:03.316 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.329+0000 7f89ff57a640 1 librados: starting msgr at 2024-03-29T10:53:03.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.329+0000 7f89ff57a640 1 librados: starting objecter 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.329+0000 7f89ff57a640 1 Processor -- start 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.330+0000 7f89ff57a640 1 -- start start 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.330+0000 7f89ff57a640 1 librados: setting wanted keys 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.330+0000 7f89ff57a640 1 librados: calling monclient init 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.330+0000 7f89ff57a640 10 monclient: init 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.330+0000 7f89ff57a640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:03.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.330+0000 7f89ff57a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:03.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 10 monclient(hunting): picked mon.noname-b con 0x7f89f80b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 10 monclient(hunting): picked mon.noname-c con 0x7f89f81422b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 10 monclient(hunting): picked mon.noname-a con 0x7f89f8142690 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f806efd0 con 0x7f89f8142690 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80b6d10 con 0x7f89f80b17d0 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f812f4f0 con 0x7f89f81422b0 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.332+0000 7f89ff57a640 10 monclient(hunting): authenticate will time out at 857.601706s 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.336+0000 7f89fd316640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f89f80b17d0 0x7f89f812ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:41401/0 (socket says 172.21.15.45:41401) 2024-03-29T10:53:03.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.336+0000 7f89fd316640 1 -- 172.21.15.45:0/4034581798 learned_addr learned my addr 172.21.15.45:0/4034581798 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 10 client.?.objecter ms_handle_connect 0x7f89f80b17d0 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968140336 0 0) 0x7f89f80b6d10 con 0x7f89f80b17d0 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.337+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89dc004eb0 con 0x7f89f80b17d0 2024-03-29T10:53:03.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1239044045 0 0) 0x7f89dc004eb0 con 0x7f89f80b17d0 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89f80b6d10 con 0x7f89f80b17d0 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89f4001960 con 0x7f89f80b17d0 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 10 monclient(hunting): dump: 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.322 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.323 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:03.323 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:03.323 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:03.323 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:03.323 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:03.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.338+0000 7f89edffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 10 client.?.objecter ms_handle_connect 0x7f89f8142690 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2001555614 0 0) 0x7f89f80b6d10 con 0x7f89f80b17d0 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 >> v1:172.21.15.45:6790/0 conn(0x7f89f81422b0 legacy=0x7f89f813d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 >> v1:172.21.15.45:6789/0 conn(0x7f89f8142690 legacy=0x7f89f813e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 1 monclient: found mon.b 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f812f680 con 0x7f89f80b17d0 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:03.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89ff57a640 5 monclient: authenticate success, global_id 4261 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89ff57a640 10 monclient: _renew_subs 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89ff57a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.339+0000 7f89ff57a640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f80b68c0 con 0x7f89f80b17d0 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.340+0000 7f89ff57a640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.340+0000 7f89ff57a640 10 monclient: _renew_subs 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.340+0000 7f89ff57a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.340+0000 7f89ff57a640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f8071b00 con 0x7f89f80b17d0 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.340+0000 7f89ff57a640 1 librados: init done 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.341+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89f4001ac0 con 0x7f89f80b17d0 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.341+0000 7f89edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.341+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89f4003a40 con 0x7f89f80b17d0 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.341+0000 7f89edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.341+0000 7f89edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.341+0000 7f89edffb640 10 monclient: dump: 2024-03-29T10:53:03.329 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.342+0000 7f89ef7fe640 4 set_mon_vals no callback set 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.342+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f89f4003c60 con 0x7f89f80b17d0 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.343+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f89f4084b00 con 0x7f89f80b17d0 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.343+0000 7f89edffb640 10 client.4261.objecter ms_dispatch 0x7f89f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:03.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.343+0000 7f89edffb640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.343+0000 7f89edffb640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.343+0000 7f89edffb640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.344+0000 7f89e37fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.344+0000 7f89fcb15640 10 monclient: get_auth_request con 0x7f89dc06dba0 auth_method 0 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.344+0000 7f89e37fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.344+0000 7f89e37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.344+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d8005080 con 0x7f89f80b17d0 2024-03-29T10:53:03.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.344+0000 7f89edffb640 10 client.4261.objecter ms_handle_connect 0x7f89dc06dba0 2024-03-29T10:53:03.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.353+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89f40503e0 con 0x7f89f80b17d0 2024-03-29T10:53:03.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.353+0000 7f89edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.353+0000 7f89edffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:03.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.686+0000 7f89e37fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.686+0000 7f89e37fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.686+0000 7f89e37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:03.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.686+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89d8005330 con 0x7f89f80b17d0 2024-03-29T10:53:03.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.687+0000 7f89edffb640 1 -- 172.21.15.45:0/4034581798 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f89f4052310 con 0x7f89f80b17d0 2024-03-29T10:53:03.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.687+0000 7f89edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.687+0000 7f89edffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:03.669 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:03.669 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:03.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 10 librados: watch_flush enter 2024-03-29T10:53:03.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 10 librados: watch_flush exit 2024-03-29T10:53:03.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T10:53:03.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 10 client.4261.objecter successfully canceled tick 2024-03-29T10:53:03.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 >> v1:172.21.15.116:6816/3290579615 conn(0x7f89dc06dba0 legacy=0x7f89dc070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 10 monclient: shutdown 2024-03-29T10:53:03.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:03.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.691+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 >> v1:172.21.15.116:6789/0 conn(0x7f89f80b17d0 legacy=0x7f89f812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.693+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 shutdown_connections 2024-03-29T10:53:03.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.693+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 >> 172.21.15.45:0/4034581798 conn(0x7f89f8066a00 msgr2=0x7f89f813b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:03.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.693+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 shutdown_connections 2024-03-29T10:53:03.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.693+0000 7f89e37fe640 1 -- 172.21.15.45:0/4034581798 wait complete. 2024-03-29T10:53:03.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:03.694+0000 7f89e37fe640 1 librados: shutdown 2024-03-29T10:53:03.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:10.507 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:10.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.684+0000 7f3e3de31640 10 monclient: get_monmap_and_config 2024-03-29T10:53:10.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.684+0000 7f3e3de31640 10 monclient: build_initial_monmap 2024-03-29T10:53:10.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.684+0000 7f3e3de31640 10 monclient: monmap: 2024-03-29T10:53:10.669 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:10.669 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:10.669 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:10.686656+0000 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:10.686656+0000 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.686+0000 7f3e3de31640 1 Processor -- start 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.686+0000 7f3e3de31640 1 -- start start 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.686+0000 7f3e3de31640 10 monclient: init 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.687+0000 7f3e3de31640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:10.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.687+0000 7f3e3de31640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:10.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.688+0000 7f3e3de31640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380a9750 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:10.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.690+0000 7f3e3ce2f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e380a9750 0x7f3e38143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42163/0 (socket says 172.21.15.45:42163) 2024-03-29T10:53:10.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.691+0000 7f3e3ce2f640 1 -- 172.21.15.45:0/1543821739 learned_addr learned my addr 172.21.15.45:0/1543821739 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:10.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.691+0000 7f3e3de31640 10 monclient(hunting): picked mon.noname-c con 0x7f3e380adca0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:10.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.691+0000 7f3e3de31640 10 monclient(hunting): picked mon.noname-b con 0x7f3e38146a80 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:10.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.692+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380abdc0 con 0x7f3e380a9750 2024-03-29T10:53:10.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.692+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b0070 con 0x7f3e380adca0 2024-03-29T10:53:10.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.692+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380af880 con 0x7f3e38146a80 2024-03-29T10:53:10.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.692+0000 7f3e3de31640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:10.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.692+0000 7f3e3de31640 10 monclient(hunting): authenticate will time out at 864.962392s 2024-03-29T10:53:10.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446847887 0 0) 0x7f3e380b0070 con 0x7f3e380adca0 2024-03-29T10:53:10.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:53:10.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e20003520 con 0x7f3e380adca0 2024-03-29T10:53:10.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141054519 0 0) 0x7f3e20003520 con 0x7f3e380adca0 2024-03-29T10:53:10.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.693+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 >> v1:172.21.15.45:6790/0 conn(0x7f3e38146a80 legacy=0x7f3e38151bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 >> v1:172.21.15.45:6789/0 conn(0x7f3e380a9750 legacy=0x7f3e38143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 1 monclient: found mon.noname-c 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380ac030 con 0x7f3e380adca0 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e2c001690 con 0x7f3e380adca0 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e3de31640 5 monclient: authenticate success, global_id 4264 2024-03-29T10:53:10.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e3de31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.695+0000 7f3e377fe640 10 monclient: dump: 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.696+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e2c001920 con 0x7f3e380adca0 2024-03-29T10:53:10.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.696+0000 7f3e3de31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.696+0000 7f3e377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.696+0000 7f3e377fe640 1 -- 172.21.15.45:0/1543821739 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e2c001cb0 con 0x7f3e380adca0 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.696+0000 7f3e3de31640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.696+0000 7f3e3de31640 4 set_mon_vals no callback set 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.697+0000 7f3e3de31640 10 monclient: shutdown 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.697+0000 7f3e377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.697+0000 7f3e377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.697+0000 7f3e377fe640 10 monclient: dump: 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:10.684 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.697+0000 7f3e3de31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.697+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 >> v1:172.21.15.116:6789/0 conn(0x7f3e380adca0 legacy=0x7f3e38146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.698+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 shutdown_connections 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.698+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 >> 172.21.15.45:0/1543821739 conn(0x7f3e380673e0 msgr2=0x7f3e380b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.698+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 shutdown_connections 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.698+0000 7f3e3de31640 1 -- 172.21.15.45:0/1543821739 wait complete. 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.699+0000 7f3e3de31640 10 monclient: build_initial_monmap 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.699+0000 7f3e3de31640 10 monclient: monmap: 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:10.685 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.699+0000 7f3e3de31640 1 librados: starting msgr at 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.699+0000 7f3e3de31640 1 librados: starting objecter 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.700+0000 7f3e3de31640 1 Processor -- start 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.700+0000 7f3e3de31640 1 -- start start 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.700+0000 7f3e3de31640 1 librados: setting wanted keys 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.700+0000 7f3e3de31640 1 librados: calling monclient init 2024-03-29T10:53:10.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.700+0000 7f3e3de31640 10 monclient: init 2024-03-29T10:53:10.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.701+0000 7f3e3de31640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:10.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.701+0000 7f3e3de31640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:10.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.705+0000 7f3e3de31640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380a9750 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:10.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 10 monclient(hunting): picked mon.noname-c con 0x7f3e380adca0 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:10.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 10 monclient(hunting): picked mon.noname-b con 0x7f3e38146a80 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:10.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e381db3d0 con 0x7f3e380a9750 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3813ebd0 con 0x7f3e38146a80 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38146300 con 0x7f3e380adca0 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.706+0000 7f3e3de31640 10 monclient(hunting): authenticate will time out at 864.976095s 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.708+0000 7f3e3ce2f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e380a9750 0x7f3e381db860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42335/0 (socket says 172.21.15.45:42335) 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.708+0000 7f3e37fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3e380adca0 0x7f3e3813ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53933/0 (socket says 172.21.15.45:53933) 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.708+0000 7f3e37fff640 1 -- 172.21.15.45:0/1997840510 learned_addr learned my addr 172.21.15.45:0/1997840510 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:10.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.708+0000 7f3e357fa640 10 client.?.objecter ms_handle_connect 0x7f3e380adca0 2024-03-29T10:53:10.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:10.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 client.?.objecter ms_handle_connect 0x7f3e380a9750 2024-03-29T10:53:10.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:10.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528617377 0 0) 0x7f3e381db3d0 con 0x7f3e380a9750 2024-03-29T10:53:10.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:53:10.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.710+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e24004eb0 con 0x7f3e380a9750 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125596490 0 0) 0x7f3e24004eb0 con 0x7f3e380a9750 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e381db3d0 con 0x7f3e380a9750 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e280018b0 con 0x7f3e380a9750 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient(hunting): dump: 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:10.694 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1745256733 0 0) 0x7f3e381db3d0 con 0x7f3e380a9750 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 >> v1:172.21.15.45:6790/0 conn(0x7f3e380adca0 legacy=0x7f3e3813ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 >> v1:172.21.15.116:6789/0 conn(0x7f3e38146a80 legacy=0x7f3e3813e4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient: _finish_hunting 0 2024-03-29T10:53:10.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 monclient: found mon.a 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e381dbf70 con 0x7f3e380a9750 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient: _finish_auth 0 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 10 monclient: _check_auth_tickets 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e3de31640 5 monclient: authenticate success, global_id 4275 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.711+0000 7f3e3de31640 10 monclient: _renew_subs 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e380b2550 con 0x7f3e380a9750 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 10 monclient: _renew_subs 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:10.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e38072700 con 0x7f3e380a9750 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e3de31640 1 librados: init done 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e28001a80 con 0x7f3e380a9750 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e28004a90 con 0x7f3e380a9750 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.712+0000 7f3e357fa640 10 monclient: dump: 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.697 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:10.698 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.698 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:10.698 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:10.698 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:10.698 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:10.698 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:10.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.713+0000 7f3e367fc640 4 set_mon_vals no callback set 2024-03-29T10:53:10.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.714+0000 7f3e3de31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.714+0000 7f3e3de31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.715+0000 7f3e3de31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:10.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.715+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e18005080 con 0x7f3e380a9750 2024-03-29T10:53:10.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.719+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f3e2801aa60 con 0x7f3e380a9750 2024-03-29T10:53:10.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.720+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f3e28084a30 con 0x7f3e380a9750 2024-03-29T10:53:10.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.720+0000 7f3e357fa640 10 client.4275.objecter ms_dispatch 0x7f3e38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:10.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.720+0000 7f3e357fa640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:10.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.720+0000 7f3e37fff640 10 monclient: get_auth_request con 0x7f3e2406dc60 auth_method 0 2024-03-29T10:53:10.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.720+0000 7f3e357fa640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:10.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.720+0000 7f3e357fa640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T10:53:10.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.721+0000 7f3e357fa640 10 client.4275.objecter ms_handle_connect 0x7f3e2406dc60 2024-03-29T10:53:10.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.728+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e28050140 con 0x7f3e380a9750 2024-03-29T10:53:10.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.728+0000 7f3e357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:10.728+0000 7f3e357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:11.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.050+0000 7f3e3de31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.050+0000 7f3e3de31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.050+0000 7f3e3de31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:11.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.050+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e18005330 con 0x7f3e380a9750 2024-03-29T10:53:11.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.051+0000 7f3e357fa640 1 -- 172.21.15.45:0/1997840510 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f3e28052240 con 0x7f3e380a9750 2024-03-29T10:53:11.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.051+0000 7f3e357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.051+0000 7f3e357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:11.036 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:11.036 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:11.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 10 librados: watch_flush enter 2024-03-29T10:53:11.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 10 librados: watch_flush exit 2024-03-29T10:53:11.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T10:53:11.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 10 client.4275.objecter successfully canceled tick 2024-03-29T10:53:11.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 >> v1:172.21.15.116:6816/3290579615 conn(0x7f3e2406dc60 legacy=0x7f3e24070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:11.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 10 monclient: shutdown 2024-03-29T10:53:11.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.056+0000 7f3e3de31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:11.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.057+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 >> v1:172.21.15.45:6789/0 conn(0x7f3e380a9750 legacy=0x7f3e381db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:11.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.058+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 shutdown_connections 2024-03-29T10:53:11.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.058+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 >> 172.21.15.45:0/1997840510 conn(0x7f3e380673e0 msgr2=0x7f3e380aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:11.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.060+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 shutdown_connections 2024-03-29T10:53:11.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.060+0000 7f3e3de31640 1 -- 172.21.15.45:0/1997840510 wait complete. 2024-03-29T10:53:11.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:11.060+0000 7f3e3de31640 1 librados: shutdown 2024-03-29T10:53:11.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:17.867 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:18.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.041+0000 7f89cfc8d640 10 monclient: get_monmap_and_config 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.041+0000 7f89cfc8d640 10 monclient: build_initial_monmap 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.041+0000 7f89cfc8d640 10 monclient: monmap: 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:18.043461+0000 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:18.043461+0000 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:18.024 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.042+0000 7f89cfc8d640 1 Processor -- start 2024-03-29T10:53:18.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.042+0000 7f89cfc8d640 1 -- start start 2024-03-29T10:53:18.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.042+0000 7f89cfc8d640 10 monclient: init 2024-03-29T10:53:18.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient(hunting): picked mon.noname-b con 0x7f89c806ab70 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient(hunting): picked mon.noname-a con 0x7f89c806af50 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient(hunting): picked mon.noname-c con 0x7f89c8141fb0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c80b0d50 con 0x7f89c806af50 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c80b1030 con 0x7f89c8141fb0 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c80afa70 con 0x7f89c806ab70 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.043+0000 7f89cfc8d640 10 monclient(hunting): authenticate will time out at 872.313869s 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89ce22a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f89c8141fb0 0x7f89c80715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:48460/0 (socket says 172.21.15.45:48460) 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89ce22a640 1 -- 172.21.15.45:0/1537988405 learned_addr learned my addr 172.21.15.45:0/1537988405 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155066473 0 0) 0x7f89c80b1030 con 0x7f89c8141fb0 2024-03-29T10:53:18.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89cca27640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89cca27640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89cca27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89cca27640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.044+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b4003520 con 0x7f89c8141fb0 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975371790 0 0) 0x7f89c80afa70 con 0x7f89c806ab70 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c80b1030 con 0x7f89c806ab70 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649004883 0 0) 0x7f89b4003520 con 0x7f89c8141fb0 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 >> v1:172.21.15.45:6790/0 conn(0x7f89c806ab70 legacy=0x7f89c80af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 >> v1:172.21.15.45:6789/0 conn(0x7f89c806af50 legacy=0x7f89c8072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: _finish_hunting 0 2024-03-29T10:53:18.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 monclient: found mon.noname-c 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c81423e0 con 0x7f89c8141fb0 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: _finish_auth 0 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: _check_auth_tickets 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89c0001a10 con 0x7f89c8141fb0 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: dump: 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.032 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cfc8d640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cfc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89c0001c30 con 0x7f89c8141fb0 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.045+0000 7f89cca27640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cca27640 1 -- 172.21.15.45:0/1537988405 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89c0001fc0 con 0x7f89c8141fb0 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cca27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cca27640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cca27640 10 monclient: dump: 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.033 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cfc8d640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cfc8d640 4 set_mon_vals no callback set 2024-03-29T10:53:18.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.046+0000 7f89cfc8d640 10 monclient: shutdown 2024-03-29T10:53:18.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.047+0000 7f89cfc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:18.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.047+0000 7f89cfc8d640 1 -- 172.21.15.45:0/1537988405 >> v1:172.21.15.116:6789/0 conn(0x7f89c8141fb0 legacy=0x7f89c80715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.047+0000 7f89cfc8d640 1 -- 172.21.15.45:0/1537988405 shutdown_connections 2024-03-29T10:53:18.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.047+0000 7f89cfc8d640 1 -- 172.21.15.45:0/1537988405 >> 172.21.15.45:0/1537988405 conn(0x7f89c8067470 msgr2=0x7f89c80b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:18.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.047+0000 7f89cfc8d640 1 -- 172.21.15.45:0/1537988405 shutdown_connections 2024-03-29T10:53:18.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.048+0000 7f89cfc8d640 1 -- 172.21.15.45:0/1537988405 wait complete. 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.049+0000 7f89cfc8d640 10 monclient: build_initial_monmap 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.049+0000 7f89cfc8d640 10 monclient: monmap: 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:18.038 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.049+0000 7f89cfc8d640 1 librados: starting msgr at 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.049+0000 7f89cfc8d640 1 librados: starting objecter 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.049+0000 7f89cfc8d640 1 Processor -- start 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.050+0000 7f89cfc8d640 1 -- start start 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.050+0000 7f89cfc8d640 1 librados: setting wanted keys 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.050+0000 7f89cfc8d640 1 librados: calling monclient init 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.050+0000 7f89cfc8d640 10 monclient: init 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient(hunting): picked mon.noname-b con 0x7f89c806ab70 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient(hunting): picked mon.noname-a con 0x7f89c8141fb0 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient(hunting): picked mon.noname-c con 0x7f89c8136860 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c8132d60 con 0x7f89c8141fb0 2024-03-29T10:53:18.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c81387e0 con 0x7f89c806ab70 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c8138a50 con 0x7f89c8136860 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cfc8d640 10 monclient(hunting): authenticate will time out at 872.321659s 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.051+0000 7f89cd228640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f89c8141fb0 0x7f89c8131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:37410/0 (socket says 172.21.15.45:37410) 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89cd228640 1 -- 172.21.15.45:0/3776604084 learned_addr learned my addr 172.21.15.45:0/3776604084 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89be7fc640 10 client.?.objecter ms_handle_connect 0x7f89c8136860 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89be7fc640 10 client.?.objecter ms_handle_connect 0x7f89c806ab70 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89be7fc640 10 client.?.objecter ms_handle_connect 0x7f89c8141fb0 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.052+0000 7f89be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400279743 0 0) 0x7f89c81387e0 con 0x7f89c806ab70 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ac004eb0 con 0x7f89c806ab70 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071373573 0 0) 0x7f89c8138a50 con 0x7f89c8136860 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c81387e0 con 0x7f89c8136860 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665806905 0 0) 0x7f89c8132d60 con 0x7f89c8141fb0 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.053+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c8138a50 con 0x7f89c8141fb0 2024-03-29T10:53:18.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2610883323 0 0) 0x7f89c81387e0 con 0x7f89c8136860 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89c8132d60 con 0x7f89c8136860 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89c0001f40 con 0x7f89c8136860 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): dump: 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2329118568 0 0) 0x7f89ac004eb0 con 0x7f89c806ab70 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89c81387e0 con 0x7f89c806ab70 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818577320 0 0) 0x7f89c8138a50 con 0x7f89c8141fb0 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89ac004eb0 con 0x7f89c8141fb0 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89b80018b0 con 0x7f89c806ab70 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): dump: 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.043 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89c4003f40 con 0x7f89c8141fb0 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 10 monclient(hunting): dump: 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:18.044 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895084790 0 0) 0x7f89c81387e0 con 0x7f89c806ab70 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.054+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 >> v1:172.21.15.45:6790/0 conn(0x7f89c8136860 legacy=0x7f89c81e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 >> v1:172.21.15.45:6789/0 conn(0x7f89c8141fb0 legacy=0x7f89c8131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 1 monclient: found mon.b 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c8136380 con 0x7f89c806ab70 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 10 monclient: _renew_subs 2024-03-29T10:53:18.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c8136fb0 con 0x7f89c806ab70 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 10 monclient: _renew_subs 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c81dbf00 con 0x7f89c806ab70 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89cfc8d640 1 librados: init done 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89b8001a10 con 0x7f89c806ab70 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f89b8003850 con 0x7f89c806ab70 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89be7fc640 10 monclient: dump: 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:18.046 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:18.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.055+0000 7f89bffff640 4 set_mon_vals no callback set 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.057+0000 7f89abfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.057+0000 7f89abfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.057+0000 7f89abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.057+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c8136570 con 0x7f89c806ab70 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.059+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f89b80039b0 con 0x7f89c806ab70 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f89b8083870 con 0x7f89c806ab70 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89be7fc640 10 client.4270.objecter ms_dispatch 0x7f89c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89cd228640 10 monclient: get_auth_request con 0x7f89ac06e150 auth_method 0 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89be7fc640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89be7fc640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89be7fc640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.060+0000 7f89be7fc640 10 client.4270.objecter ms_handle_connect 0x7f89ac06e150 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.066+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89b804ef80 con 0x7f89c806ab70 2024-03-29T10:53:18.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.066+0000 7f89be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.067+0000 7f89be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:18.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.490+0000 7f89abfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.490+0000 7f89abfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.490+0000 7f89abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:18.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.490+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89c81db430 con 0x7f89c806ab70 2024-03-29T10:53:18.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.491+0000 7f89be7fc640 1 -- 172.21.15.45:0/3776604084 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f89b8051080 con 0x7f89c806ab70 2024-03-29T10:53:18.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.491+0000 7f89be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.491+0000 7f89be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:18.474 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:18.474 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:18.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.495+0000 7f89abfff640 10 librados: watch_flush enter 2024-03-29T10:53:18.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.495+0000 7f89abfff640 10 librados: watch_flush exit 2024-03-29T10:53:18.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.495+0000 7f89abfff640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:53:18.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.495+0000 7f89abfff640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:53:18.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 >> v1:172.21.15.116:6816/3290579615 conn(0x7f89ac06e150 legacy=0x7f89ac070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 10 monclient: shutdown 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 >> v1:172.21.15.116:6789/0 conn(0x7f89c806ab70 legacy=0x7f89c81db7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 shutdown_connections 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 >> 172.21.15.45:0/3776604084 conn(0x7f89c8067470 msgr2=0x7f89c80b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 shutdown_connections 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 -- 172.21.15.45:0/3776604084 wait complete. 2024-03-29T10:53:18.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:18.496+0000 7f89abfff640 1 librados: shutdown 2024-03-29T10:53:18.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:25.302 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.454+0000 7f453e495640 10 monclient: get_monmap_and_config 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.454+0000 7f453e495640 10 monclient: build_initial_monmap 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.455+0000 7f453e495640 10 monclient: monmap: 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:25.455832+0000 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:25.455832+0000 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:25.436 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:25.437 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:25.437 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:25.437 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:25.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.456+0000 7f453e495640 1 Processor -- start 2024-03-29T10:53:25.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.456+0000 7f453e495640 1 -- start start 2024-03-29T10:53:25.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.456+0000 7f453e495640 10 monclient: init 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient(hunting): picked mon.noname-b con 0x7f45380b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient(hunting): picked mon.noname-c con 0x7f4538071eb0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient(hunting): picked mon.noname-a con 0x7f4538143d10 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453806ef80 con 0x7f4538143d10 2024-03-29T10:53:25.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453806f260 con 0x7f4538071eb0 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45380b40c0 con 0x7f45380b1f60 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f453e495640 10 monclient(hunting): authenticate will time out at 879.726418s 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f4537fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f45380b1f60 0x7f4538143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59712/0 (socket says 172.21.15.45:59712) 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.457+0000 7f4537fff640 1 -- 172.21.15.45:0/684421900 learned_addr learned my addr 172.21.15.45:0/684421900 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483385951 0 0) 0x7f45380b40c0 con 0x7f45380b1f60 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4524003520 con 0x7f45380b1f60 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703721096 0 0) 0x7f4524003520 con 0x7f45380b1f60 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 >> v1:172.21.15.116:6789/0 conn(0x7f4538071eb0 legacy=0x7f4538146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:25.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.458+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 >> v1:172.21.15.45:6789/0 conn(0x7f4538143d10 legacy=0x7f4538146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 1 monclient: found mon.noname-b 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4538155d70 con 0x7f45380b1f60 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:25.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f453e495640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f453e495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4520001690 con 0x7f45380b1f60 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: dump: 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:25.444 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f453e495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4520001860 con 0x7f45380b1f60 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f45377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f453e495640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.459+0000 7f453e495640 4 set_mon_vals no callback set 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f453e495640 10 monclient: shutdown 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f453e495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f453e495640 1 -- 172.21.15.45:0/684421900 >> v1:172.21.15.45:6790/0 conn(0x7f45380b1f60 legacy=0x7f4538143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f45377fe640 1 -- 172.21.15.45:0/684421900 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4520001cb0 con 0x7f45380b1f60 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f45377fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f453e495640 1 -- 172.21.15.45:0/684421900 shutdown_connections 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.460+0000 7f453e495640 1 -- 172.21.15.45:0/684421900 >> 172.21.15.45:0/684421900 conn(0x7f4538066a00 msgr2=0x7f4538066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:25.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.461+0000 7f453e495640 1 -- 172.21.15.45:0/684421900 shutdown_connections 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.461+0000 7f453e495640 1 -- 172.21.15.45:0/684421900 wait complete. 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 10 monclient: build_initial_monmap 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 10 monclient: monmap: 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:25.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 1 librados: starting msgr at 2024-03-29T10:53:25.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 1 librados: starting objecter 2024-03-29T10:53:25.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 1 Processor -- start 2024-03-29T10:53:25.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 1 -- start start 2024-03-29T10:53:25.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 1 librados: setting wanted keys 2024-03-29T10:53:25.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 1 librados: calling monclient init 2024-03-29T10:53:25.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.462+0000 7f453e495640 10 monclient: init 2024-03-29T10:53:25.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.463+0000 7f453e495640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.463+0000 7f453e495640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.463+0000 7f453e495640 10 monclient(hunting): picked mon.noname-c con 0x7f45380b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.463+0000 7f453e495640 10 monclient(hunting): picked mon.noname-a con 0x7f4538143d10 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f4537fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f45380b1f60 0x7f4538131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60978/0 (socket says 172.21.15.45:60978) 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f453e495640 10 monclient(hunting): picked mon.noname-b con 0x7f4538153530 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f4537fff640 1 -- 172.21.15.45:0/3770178492 learned_addr learned my addr 172.21.15.45:0/3770178492 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f45357fa640 10 client.?.objecter ms_handle_connect 0x7f4538153530 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f45357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f453e495640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4538142bd0 con 0x7f4538143d10 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f45357fa640 10 client.?.objecter ms_handle_connect 0x7f45380b1f60 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f45357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f453e495640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4538143060 con 0x7f4538153530 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f453e495640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4538132740 con 0x7f45380b1f60 2024-03-29T10:53:25.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f453e495640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.464+0000 7f453e495640 10 monclient(hunting): authenticate will time out at 879.733828s 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882032824 0 0) 0x7f4538132740 con 0x7f45380b1f60 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f45380b1f60 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832999895 0 0) 0x7f4538143060 con 0x7f4538153530 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4538132740 con 0x7f4538153530 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087066648 0 0) 0x7f4538132740 con 0x7f4538153530 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4538143060 con 0x7f4538153530 2024-03-29T10:53:25.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4530001a90 con 0x7f4538153530 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): dump: 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:25.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2170000267 0 0) 0x7f4518004eb0 con 0x7f45380b1f60 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4538132740 con 0x7f45380b1f60 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4520001bb0 con 0x7f45380b1f60 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.465+0000 7f45357fa640 10 monclient(hunting): dump: 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:25.458 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.466+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2991465052 0 0) 0x7f4538132740 con 0x7f45380b1f60 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.466+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 >> v1:172.21.15.116:6789/0 conn(0x7f4538153530 legacy=0x7f4538132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.466+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 >> v1:172.21.15.45:6789/0 conn(0x7f4538143d10 legacy=0x7f45381340e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: _finish_hunting 0 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 1 monclient: found mon.c 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4538142310 con 0x7f45380b1f60 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: _finish_auth 0 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: _check_auth_tickets 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 10 monclient: _renew_subs 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:25.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45381425f0 con 0x7f45380b1f60 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 10 monclient: _renew_subs 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4538071310 con 0x7f45380b1f60 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f453e495640 1 librados: init done 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4520004bf0 con 0x7f45380b1f60 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4520004f00 con 0x7f45380b1f60 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f45357fa640 10 monclient: dump: 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:25.460 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.467+0000 7f4536ffd640 4 set_mon_vals no callback set 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.468+0000 7f452e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.469+0000 7f452e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.469+0000 7f452e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.469+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4538075f10 con 0x7f45380b1f60 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.469+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f452001aed0 con 0x7f45380b1f60 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.470+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f4520084bb0 con 0x7f45380b1f60 2024-03-29T10:53:25.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.470+0000 7f45357fa640 10 client.4286.objecter ms_dispatch 0x7f4538000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.470+0000 7f45357fa640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.470+0000 7f45357fa640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.470+0000 7f45357fa640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.470+0000 7f452f7fe640 10 monclient: get_auth_request con 0x7f451806ded0 auth_method 0 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.471+0000 7f45357fa640 10 client.4286.objecter ms_handle_connect 0x7f451806ded0 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.477+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4520050490 con 0x7f45380b1f60 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.478+0000 7f45357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.478+0000 7f45357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:25.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f452e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:25.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f452e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:25.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f452e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:25.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45380b7940 con 0x7f45380b1f60 2024-03-29T10:53:25.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f45357fa640 1 -- 172.21.15.45:0/3770178492 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f45200523c0 con 0x7f45380b1f60 2024-03-29T10:53:25.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f45357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:25.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.809+0000 7f45357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:25.791 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:25.792 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:25.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 10 librados: watch_flush enter 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 10 librados: watch_flush exit 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 10 client.4286.objecter successfully canceled tick 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 >> v1:172.21.15.116:6816/3290579615 conn(0x7f451806ded0 legacy=0x7f4518070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 10 monclient: shutdown 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 >> v1:172.21.15.45:6790/0 conn(0x7f45380b1f60 legacy=0x7f4538131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 shutdown_connections 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 >> 172.21.15.45:0/3770178492 conn(0x7f4538066a00 msgr2=0x7f45380b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 shutdown_connections 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.813+0000 7f452e7fc640 1 -- 172.21.15.45:0/3770178492 wait complete. 2024-03-29T10:53:25.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:25.814+0000 7f452e7fc640 1 librados: shutdown 2024-03-29T10:53:25.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:32.616 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:32.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.784+0000 7f2391c3e640 10 monclient: get_monmap_and_config 2024-03-29T10:53:32.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.784+0000 7f2391c3e640 10 monclient: build_initial_monmap 2024-03-29T10:53:32.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.784+0000 7f2391c3e640 10 monclient: monmap: 2024-03-29T10:53:32.765 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:32.785194+0000 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:32.785194+0000 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:32.766 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.785+0000 7f2391c3e640 1 Processor -- start 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.785+0000 7f2391c3e640 1 -- start start 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.785+0000 7f2391c3e640 10 monclient: init 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.786+0000 7f2391c3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.786+0000 7f2391c3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.786+0000 7f2391c3e640 10 monclient(hunting): picked mon.noname-b con 0x7f238c1457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:32.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.786+0000 7f2391c3e640 10 monclient(hunting): picked mon.noname-c con 0x7f238c13f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:32.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.788+0000 7f238affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f238c13f650 0x7f238c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:41492/0 (socket says 172.21.15.45:41492) 2024-03-29T10:53:32.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.788+0000 7f238affd640 1 -- 172.21.15.45:0/1501299769 learned_addr learned my addr 172.21.15.45:0/1501299769 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:32.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.788+0000 7f2391c3e640 10 monclient(hunting): picked mon.noname-a con 0x7f238c0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:32.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.788+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0aaef0 con 0x7f238c0a9d70 2024-03-29T10:53:32.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c145bd0 con 0x7f238c13f650 2024-03-29T10:53:32.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c145eb0 con 0x7f238c1457f0 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f2391c3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f2391c3e640 10 monclient(hunting): authenticate will time out at 887.058145s 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956753185 0 0) 0x7f238c0aaef0 con 0x7f238c0a9d70 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378003520 con 0x7f238c0a9d70 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975289108 0 0) 0x7f238c145bd0 con 0x7f238c13f650 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c0aaef0 con 0x7f238c13f650 2024-03-29T10:53:32.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3824702833 0 0) 0x7f2378003520 con 0x7f238c0a9d70 2024-03-29T10:53:32.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.789+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 >> v1:172.21.15.45:6790/0 conn(0x7f238c1457f0 legacy=0x7f238c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:32.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 >> v1:172.21.15.116:6789/0 conn(0x7f238c13f650 legacy=0x7f238c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:32.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:32.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:53:32.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c146340 con 0x7f238c0a9d70 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2384001a10 con 0x7f238c0a9d70 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: dump: 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:32.773 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.774 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.774 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:32.774 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.774 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:32.774 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f2391c3e640 5 monclient: authenticate success, global_id 4281 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f2391c3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2384001b70 con 0x7f238c0a9d70 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 1 -- 172.21.15.45:0/1501299769 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2384001f80 con 0x7f238c0a9d70 2024-03-29T10:53:32.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f238a7fc640 10 monclient: dump: 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f2391c3e640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:32.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.791+0000 7f2391c3e640 4 set_mon_vals no callback set 2024-03-29T10:53:32.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.792+0000 7f2391c3e640 10 monclient: shutdown 2024-03-29T10:53:32.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.792+0000 7f2391c3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:32.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.792+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 >> v1:172.21.15.45:6789/0 conn(0x7f238c0a9d70 legacy=0x7f238c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.793+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 shutdown_connections 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.793+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 >> 172.21.15.45:0/1501299769 conn(0x7f238c066a00 msgr2=0x7f238c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.793+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 shutdown_connections 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.794+0000 7f2391c3e640 1 -- 172.21.15.45:0/1501299769 wait complete. 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.794+0000 7f2391c3e640 10 monclient: build_initial_monmap 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.794+0000 7f2391c3e640 10 monclient: monmap: 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:32.779 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.795+0000 7f2391c3e640 1 librados: starting msgr at 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.795+0000 7f2391c3e640 1 librados: starting objecter 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.795+0000 7f2391c3e640 1 Processor -- start 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.795+0000 7f2391c3e640 1 -- start start 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.795+0000 7f2391c3e640 1 librados: setting wanted keys 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.795+0000 7f2391c3e640 1 librados: calling monclient init 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.796+0000 7f2391c3e640 10 monclient: init 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.796+0000 7f2391c3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.797+0000 7f2391c3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:32.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.799+0000 7f2391c3e640 10 monclient(hunting): picked mon.noname-c con 0x7f238c0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 10 monclient(hunting): picked mon.noname-a con 0x7f238c13f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 10 monclient(hunting): picked mon.noname-b con 0x7f238c1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c137990 con 0x7f238c13f650 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0ac790 con 0x7f238c1457f0 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c142080 con 0x7f238c0a9d70 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f2391c3e640 10 monclient(hunting): authenticate will time out at 887.069266s 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f238b7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f238c0a9d70 0x7f238c1425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:42833/0 (socket says 172.21.15.45:42833) 2024-03-29T10:53:32.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.800+0000 7f238b7fe640 1 -- 172.21.15.45:0/237341473 learned_addr learned my addr 172.21.15.45:0/237341473 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:32.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.802+0000 7f2383fff640 10 client.?.objecter ms_handle_connect 0x7f238c1457f0 2024-03-29T10:53:32.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.802+0000 7f2383fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:32.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 client.?.objecter ms_handle_connect 0x7f238c13f650 2024-03-29T10:53:32.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:32.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 client.?.objecter ms_handle_connect 0x7f238c0a9d70 2024-03-29T10:53:32.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921946139 0 0) 0x7f238c137990 con 0x7f238c13f650 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f236c004eb0 con 0x7f238c13f650 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740571963 0 0) 0x7f238c142080 con 0x7f238c0a9d70 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:53:32.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c137990 con 0x7f238c0a9d70 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.803+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646503957 0 0) 0x7f236c004eb0 con 0x7f238c13f650 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f238c142080 con 0x7f238c13f650 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f237c001930 con 0x7f238c13f650 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): dump: 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:32.787 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403098604 0 0) 0x7f238c137990 con 0x7f238c0a9d70 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f236c004eb0 con 0x7f238c0a9d70 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2374001750 con 0x7f238c0a9d70 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient(hunting): dump: 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.788 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3514954081 0 0) 0x7f238c142080 con 0x7f238c13f650 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 >> v1:172.21.15.45:6790/0 conn(0x7f238c0a9d70 legacy=0x7f238c1425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 >> v1:172.21.15.116:6789/0 conn(0x7f238c1457f0 legacy=0x7f238c137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 monclient: found mon.a 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c142ce0 con 0x7f238c13f650 2024-03-29T10:53:32.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2383fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 10 monclient: _renew_subs 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c0ab600 con 0x7f238c13f650 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 10 monclient: _renew_subs 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c13ff80 con 0x7f238c13f650 2024-03-29T10:53:32.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.804+0000 7f2391c3e640 1 librados: init done 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.806+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f237c0047e0 con 0x7f238c13f650 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2383fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f237c004af0 con 0x7f238c13f650 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2383fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2383fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2383fff640 10 monclient: dump: 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.792 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f237c01aac0 con 0x7f238c13f650 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2391c3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2391c3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2391c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.807+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c075f10 con 0x7f238c13f650 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f238b7fe640 10 monclient: get_auth_request con 0x7f236c06df30 auth_method 0 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2389ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f237c084ac0 con 0x7f238c13f650 2024-03-29T10:53:32.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2383fff640 10 client.4284.objecter ms_dispatch 0x7f238c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:32.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2383fff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:32.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2383fff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:32.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2383fff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T10:53:32.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.808+0000 7f2383fff640 10 client.4284.objecter ms_handle_connect 0x7f236c06df30 2024-03-29T10:53:32.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.817+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f237c050420 con 0x7f238c13f650 2024-03-29T10:53:32.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.817+0000 7f2383fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:32.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:32.817+0000 7f2383fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:33.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.148+0000 7f2391c3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.148+0000 7f2391c3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.148+0000 7f2391c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:33.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.148+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f238c0b3680 con 0x7f238c13f650 2024-03-29T10:53:33.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.150+0000 7f2383fff640 1 -- 172.21.15.45:0/237341473 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f237c052350 con 0x7f238c13f650 2024-03-29T10:53:33.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.150+0000 7f2383fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.150+0000 7f2383fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:33.132 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:33.132 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:33.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.154+0000 7f2391c3e640 10 librados: watch_flush enter 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 10 librados: watch_flush exit 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 10 client.4284.objecter successfully canceled tick 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 >> v1:172.21.15.116:6816/3290579615 conn(0x7f236c06df30 legacy=0x7f236c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 10 monclient: shutdown 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:33.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 >> v1:172.21.15.45:6789/0 conn(0x7f238c13f650 legacy=0x7f238c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 shutdown_connections 2024-03-29T10:53:33.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 >> 172.21.15.45:0/237341473 conn(0x7f238c066a00 msgr2=0x7f238c13aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:33.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.155+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 shutdown_connections 2024-03-29T10:53:33.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.156+0000 7f2391c3e640 1 -- 172.21.15.45:0/237341473 wait complete. 2024-03-29T10:53:33.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:33.156+0000 7f2391c3e640 1 librados: shutdown 2024-03-29T10:53:33.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:39.959 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.129+0000 7fac9e47b640 10 monclient: get_monmap_and_config 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.129+0000 7fac9e47b640 10 monclient: build_initial_monmap 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.129+0000 7fac9e47b640 10 monclient: monmap: 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:40.130191+0000 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:40.130191+0000 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:40.113 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:40.114 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:40.114 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:40.114 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.131+0000 7fac9e47b640 1 Processor -- start 2024-03-29T10:53:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.132+0000 7fac9e47b640 1 -- start start 2024-03-29T10:53:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.132+0000 7fac9e47b640 10 monclient: init 2024-03-29T10:53:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.132+0000 7fac9e47b640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.132+0000 7fac9e47b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.132+0000 7fac9e47b640 10 monclient(hunting): picked mon.noname-b con 0x7fac980ab290 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.132+0000 7fac9e47b640 10 monclient(hunting): picked mon.noname-a con 0x7fac980ab670 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9e47b640 10 monclient(hunting): picked mon.noname-c con 0x7fac980ae290 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9e47b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac980ac160 con 0x7fac980ab670 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9e47b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac98142160 con 0x7fac980ae290 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9e47b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac980ac3d0 con 0x7fac980ab290 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9ca18640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fac980ae290 0x7fac98149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:34056/0 (socket says 172.21.15.45:34056) 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9ca18640 1 -- 172.21.15.45:0/747430606 learned_addr learned my addr 172.21.15.45:0/747430606 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9e47b640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac9e47b640 10 monclient(hunting): authenticate will time out at 894.403850s 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852145388 0 0) 0x7fac98142160 con 0x7fac980ae290 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac96ffd640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:53:40.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.134+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac88003440 con 0x7fac980ae290 2024-03-29T10:53:40.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.135+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 505063273 0 0) 0x7fac88003440 con 0x7fac980ae290 2024-03-29T10:53:40.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.135+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 >> v1:172.21.15.45:6790/0 conn(0x7fac980ab290 legacy=0x7fac98141340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:40.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.135+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 >> v1:172.21.15.45:6789/0 conn(0x7fac980ab670 legacy=0x7fac98141a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 1 monclient: found mon.noname-c 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac980ac6b0 con 0x7fac980ae290 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:40.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fac90001a40 con 0x7fac980ae290 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: dump: 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:40.120 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac9e47b640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac9e47b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac90001c10 con 0x7fac980ae290 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 1 -- 172.21.15.45:0/747430606 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fac90002060 con 0x7fac980ae290 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac96ffd640 10 monclient: dump: 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:40.121 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac9e47b640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.137+0000 7fac9e47b640 4 set_mon_vals no callback set 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.138+0000 7fac9e47b640 10 monclient: shutdown 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.138+0000 7fac9e47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.138+0000 7fac9e47b640 1 -- 172.21.15.45:0/747430606 >> v1:172.21.15.116:6789/0 conn(0x7fac980ae290 legacy=0x7fac98149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.138+0000 7fac9e47b640 1 -- 172.21.15.45:0/747430606 shutdown_connections 2024-03-29T10:53:40.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.138+0000 7fac9e47b640 1 -- 172.21.15.45:0/747430606 >> 172.21.15.45:0/747430606 conn(0x7fac980b3c80 msgr2=0x7fac980b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:40.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.139+0000 7fac9e47b640 1 -- 172.21.15.45:0/747430606 shutdown_connections 2024-03-29T10:53:40.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.139+0000 7fac9e47b640 1 -- 172.21.15.45:0/747430606 wait complete. 2024-03-29T10:53:40.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.140+0000 7fac9e47b640 10 monclient: build_initial_monmap 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.140+0000 7fac9e47b640 10 monclient: monmap: 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.140+0000 7fac9e47b640 1 librados: starting msgr at 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.141+0000 7fac9e47b640 1 librados: starting objecter 2024-03-29T10:53:40.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.141+0000 7fac9e47b640 1 Processor -- start 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.141+0000 7fac9e47b640 1 -- start start 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.141+0000 7fac9e47b640 1 librados: setting wanted keys 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.141+0000 7fac9e47b640 1 librados: calling monclient init 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.141+0000 7fac9e47b640 10 monclient: init 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 10 monclient(hunting): picked mon.noname-b con 0x7fac980ab290 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 10 monclient(hunting): picked mon.noname-c con 0x7fac980ae290 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 10 monclient(hunting): picked mon.noname-a con 0x7fac98139280 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac981365e0 con 0x7fac98139280 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9813d540 con 0x7fac980ab290 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9813cbf0 con 0x7fac980ae290 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.142+0000 7fac9e47b640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:40.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.143+0000 7fac9e47b640 10 monclient(hunting): authenticate will time out at 894.412035s 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.145+0000 7fac977fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fac980ae290 0x7fac9813e3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59599/0 (socket says 172.21.15.45:59599) 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.145+0000 7fac977fe640 1 -- 172.21.15.45:0/759883420 learned_addr learned my addr 172.21.15.45:0/759883420 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 client.?.objecter ms_handle_connect 0x7fac980ae290 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 client.?.objecter ms_handle_connect 0x7fac980ab290 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454443929 0 0) 0x7fac9813cbf0 con 0x7fac980ae290 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.146+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c004eb0 con 0x7fac980ae290 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 client.?.objecter ms_handle_connect 0x7fac98139280 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966238329 0 0) 0x7fac9813d540 con 0x7fac980ab290 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac9813cbf0 con 0x7fac980ab290 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1270343641 0 0) 0x7fac7c004eb0 con 0x7fac980ae290 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac9813d540 con 0x7fac980ae290 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fac84001960 con 0x7fac980ae290 2024-03-29T10:53:40.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:40.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818700877 0 0) 0x7fac981365e0 con 0x7fac98139280 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c004eb0 con 0x7fac98139280 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1006813165 0 0) 0x7fac9813d540 con 0x7fac980ae290 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 >> v1:172.21.15.116:6789/0 conn(0x7fac980ab290 legacy=0x7fac9813dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 >> v1:172.21.15.45:6789/0 conn(0x7fac98139280 legacy=0x7fac981dd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 monclient: found mon.c 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9806e0b0 con 0x7fac980ae290 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:40.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 10 monclient: _renew_subs 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9806e390 con 0x7fac980ae290 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 10 monclient: _renew_subs 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9813d790 con 0x7fac980ae290 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac9e47b640 1 librados: init done 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac840035f0 con 0x7fac980ae290 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fac84003940 con 0x7fac980ae290 2024-03-29T10:53:40.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac95ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.147+0000 7fac94ff9640 10 monclient: dump: 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.148+0000 7fac9e47b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.148+0000 7fac9e47b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.148+0000 7fac9e47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:40.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.148+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac78005080 con 0x7fac980ae290 2024-03-29T10:53:40.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.150+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fac84003aa0 con 0x7fac980ae290 2024-03-29T10:53:40.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.151+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fac84084900 con 0x7fac980ae290 2024-03-29T10:53:40.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.151+0000 7fac94ff9640 10 client.4292.objecter ms_dispatch 0x7fac98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:40.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.151+0000 7fac94ff9640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:40.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.151+0000 7fac94ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:40.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.151+0000 7fac94ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T10:53:40.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.152+0000 7fac97fff640 10 monclient: get_auth_request con 0x7fac7c06ddf0 auth_method 0 2024-03-29T10:53:40.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.153+0000 7fac94ff9640 10 client.4292.objecter ms_handle_connect 0x7fac7c06ddf0 2024-03-29T10:53:40.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.159+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac84050240 con 0x7fac980ae290 2024-03-29T10:53:40.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.159+0000 7fac94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.159+0000 7fac94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:40.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.499+0000 7fac9e47b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:40.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.499+0000 7fac9e47b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:40.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.499+0000 7fac9e47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:53:40.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.499+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac78005330 con 0x7fac980ae290 2024-03-29T10:53:40.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.501+0000 7fac94ff9640 1 -- 172.21.15.45:0/759883420 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fac84050240 con 0x7fac980ae290 2024-03-29T10:53:40.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.501+0000 7fac94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:40.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.501+0000 7fac94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:40.483 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:40.483 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:40.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.505+0000 7fac9e47b640 10 librados: watch_flush enter 2024-03-29T10:53:40.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.505+0000 7fac9e47b640 10 librados: watch_flush exit 2024-03-29T10:53:40.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.505+0000 7fac9e47b640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T10:53:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.505+0000 7fac9e47b640 10 client.4292.objecter successfully canceled tick 2024-03-29T10:53:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.506+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 >> v1:172.21.15.116:6816/3290579615 conn(0x7fac7c06ddf0 legacy=0x7fac7c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.506+0000 7fac9e47b640 10 monclient: shutdown 2024-03-29T10:53:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.506+0000 7fac9e47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.506+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 >> v1:172.21.15.45:6790/0 conn(0x7fac980ae290 legacy=0x7fac9813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.507+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 shutdown_connections 2024-03-29T10:53:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.507+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 >> 172.21.15.45:0/759883420 conn(0x7fac980b3c80 msgr2=0x7fac9813adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.507+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 shutdown_connections 2024-03-29T10:53:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.507+0000 7fac9e47b640 1 -- 172.21.15.45:0/759883420 wait complete. 2024-03-29T10:53:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:40.507+0000 7fac9e47b640 1 librados: shutdown 2024-03-29T10:53:40.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:47.312 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:47.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.487+0000 7f139c7b6640 10 monclient: get_monmap_and_config 2024-03-29T10:53:47.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.487+0000 7f139c7b6640 10 monclient: build_initial_monmap 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.487+0000 7f139c7b6640 10 monclient: monmap: 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:47.488377+0000 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:47.488377+0000 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:47.469 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.488+0000 7f139c7b6640 1 Processor -- start 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.489+0000 7f139c7b6640 1 -- start start 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.489+0000 7f139c7b6640 10 monclient: init 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f1394074650 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f1394072e80 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f139406f510 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139406f080 con 0x7f1394074650 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1394070900 con 0x7f139406f510 2024-03-29T10:53:47.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1394143d10 con 0x7f1394072e80 2024-03-29T10:53:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.490+0000 7f139c7b6640 10 monclient(hunting): authenticate will time out at 901.759214s 2024-03-29T10:53:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.494+0000 7f139a552640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1394074650 0x7f1394143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48467/0 (socket says 172.21.15.45:48467) 2024-03-29T10:53:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.494+0000 7f139a552640 1 -- 172.21.15.45:0/1453379218 learned_addr learned my addr 172.21.15.45:0/1453379218 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534209688 0 0) 0x7f139406f080 con 0x7f1394074650 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1380003520 con 0x7f1394074650 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153940860 0 0) 0x7f1394070900 con 0x7f139406f510 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139406f080 con 0x7f139406f510 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770726898 0 0) 0x7f1394143d10 con 0x7f1394072e80 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1394070900 con 0x7f1394072e80 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965397917 0 0) 0x7f1380003520 con 0x7f1394074650 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.495+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 >> v1:172.21.15.45:6790/0 conn(0x7f1394072e80 legacy=0x7f13941499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 >> v1:172.21.15.116:6789/0 conn(0x7f139406f510 legacy=0x7f1394151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: _finish_hunting 0 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 1 monclient: found mon.noname-a 2024-03-29T10:53:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139406e8d0 con 0x7f1394074650 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: _finish_auth 0 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: _check_auth_tickets 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f1384001690 con 0x7f1394074650 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: dump: 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:47.480 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13840017f0 con 0x7f1394074650 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 1 -- 172.21.15.45:0/1453379218 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f1384001c00 con 0x7f1394074650 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:47.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.496+0000 7f1399550640 10 monclient: dump: 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:47.482 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.497+0000 7f139c7b6640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.497+0000 7f139c7b6640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.497+0000 7f139c7b6640 4 set_mon_vals no callback set 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.497+0000 7f139c7b6640 10 monclient: shutdown 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.497+0000 7f139c7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.497+0000 7f139c7b6640 1 -- 172.21.15.45:0/1453379218 >> v1:172.21.15.45:6789/0 conn(0x7f1394074650 legacy=0x7f1394143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.500+0000 7f139c7b6640 1 -- 172.21.15.45:0/1453379218 shutdown_connections 2024-03-29T10:53:47.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.500+0000 7f139c7b6640 1 -- 172.21.15.45:0/1453379218 >> 172.21.15.45:0/1453379218 conn(0x7f13940b4200 msgr2=0x7f13940b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:47.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.502+0000 7f139c7b6640 1 -- 172.21.15.45:0/1453379218 shutdown_connections 2024-03-29T10:53:47.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.504+0000 7f139c7b6640 1 -- 172.21.15.45:0/1453379218 wait complete. 2024-03-29T10:53:47.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.505+0000 7f139c7b6640 10 monclient: build_initial_monmap 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.505+0000 7f139c7b6640 10 monclient: monmap: 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 1 librados: starting msgr at 2024-03-29T10:53:47.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 1 librados: starting objecter 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 1 Processor -- start 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 1 -- start start 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 1 librados: setting wanted keys 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 1 librados: calling monclient init 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.506+0000 7f139c7b6640 10 monclient: init 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f139406f510 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:47.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f1394074650 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:47.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f13941cf540 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:47.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139413ff60 con 0x7f1394074650 2024-03-29T10:53:47.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139413d560 con 0x7f13941cf540 2024-03-29T10:53:47.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13941cf9e0 con 0x7f139406f510 2024-03-29T10:53:47.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:47.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.507+0000 7f139c7b6640 10 monclient(hunting): authenticate will time out at 901.776208s 2024-03-29T10:53:47.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.513+0000 7f1399d51640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1394074650 0x7f13941403f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48783/0 (socket says 172.21.15.45:48783) 2024-03-29T10:53:47.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.513+0000 7f1399d51640 1 -- 172.21.15.45:0/4245139000 learned_addr learned my addr 172.21.15.45:0/4245139000 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 client.?.objecter ms_handle_connect 0x7f1394074650 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 client.?.objecter ms_handle_connect 0x7f13941cf540 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641618097 0 0) 0x7f139413ff60 con 0x7f1394074650 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:53:47.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1378004eb0 con 0x7f1394074650 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519445184 0 0) 0x7f139413d560 con 0x7f13941cf540 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139413ff60 con 0x7f13941cf540 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968317717 0 0) 0x7f139413ff60 con 0x7f13941cf540 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f139413d560 con 0x7f13941cf540 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f138c001c30 con 0x7f13941cf540 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): dump: 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:47.497 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015762753 0 0) 0x7f1378004eb0 con 0x7f1394074650 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f139413ff60 con 0x7f1394074650 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f1390001a10 con 0x7f1394074650 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.514+0000 7f138affd640 10 monclient(hunting): dump: 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.515+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224488888 0 0) 0x7f139413ff60 con 0x7f1394074650 2024-03-29T10:53:47.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.515+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 >> v1:172.21.15.45:6790/0 conn(0x7f139406f510 legacy=0x7f13941d06a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 >> v1:172.21.15.116:6789/0 conn(0x7f13941cf540 legacy=0x7f13941f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 1 monclient: found mon.a 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13941cfcc0 con 0x7f1394074650 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 10 monclient: _finish_auth 0 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f138affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:47.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f139c7b6640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f139c7b6640 10 monclient: _renew_subs 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f139c7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.516+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13941cffa0 con 0x7f1394074650 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.517+0000 7f139c7b6640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.517+0000 7f139c7b6640 10 monclient: _renew_subs 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.517+0000 7f139c7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.517+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1394146e20 con 0x7f1394074650 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.517+0000 7f139c7b6640 1 librados: init done 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.518+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1390001b70 con 0x7f1394074650 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.518+0000 7f138affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.518+0000 7f138bfff640 4 set_mon_vals no callback set 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.518+0000 7f139c7b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.518+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f1390003a60 con 0x7f1394074650 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.519+0000 7f139c7b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:47.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.519+0000 7f139c7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.519+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c005080 con 0x7f1394074650 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.519+0000 7f138affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.519+0000 7f138affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.519+0000 7f138affd640 10 monclient: dump: 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:47.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.520+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f1390003bc0 con 0x7f1394074650 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.521+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f1390084a80 con 0x7f1394074650 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.521+0000 7f138affd640 10 client.4293.objecter ms_dispatch 0x7f1394000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.521+0000 7f138affd640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.521+0000 7f138affd640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.522+0000 7f138affd640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.522+0000 7f139a552640 10 monclient: get_auth_request con 0x7f137806df40 auth_method 0 2024-03-29T10:53:47.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.522+0000 7f138affd640 10 client.4293.objecter ms_handle_connect 0x7f137806df40 2024-03-29T10:53:47.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.529+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1390050360 con 0x7f1394074650 2024-03-29T10:53:47.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.529+0000 7f138affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:47.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.529+0000 7f138affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.861+0000 7f139c7b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.861+0000 7f139c7b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.861+0000 7f139c7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.861+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f137c005330 con 0x7f1394074650 2024-03-29T10:53:47.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.862+0000 7f138affd640 1 -- 172.21.15.45:0/4245139000 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f1390052290 con 0x7f1394074650 2024-03-29T10:53:47.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.862+0000 7f138affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:47.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.862+0000 7f138affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:47.850 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:47.850 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 10 librados: watch_flush enter 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 10 librados: watch_flush exit 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 10 client.4293.objecter successfully canceled tick 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 >> v1:172.21.15.116:6816/3290579615 conn(0x7f137806df40 legacy=0x7f1378070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 10 monclient: shutdown 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:47.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.869+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 >> v1:172.21.15.45:6789/0 conn(0x7f1394074650 legacy=0x7f13941403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.870+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 shutdown_connections 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.870+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 >> 172.21.15.45:0/4245139000 conn(0x7f13940b4200 msgr2=0x7f13940b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.870+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 shutdown_connections 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.870+0000 7f139c7b6640 1 -- 172.21.15.45:0/4245139000 wait complete. 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:47.870+0000 7f139c7b6640 1 librados: shutdown 2024-03-29T10:53:47.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:54.678 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.847+0000 7f2e0ac38640 10 monclient: get_monmap_and_config 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.847+0000 7f2e0ac38640 10 monclient: build_initial_monmap 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.847+0000 7f2e0ac38640 10 monclient: monmap: 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:53:54.848763+0000 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:53:54.848763+0000 2024-03-29T10:53:54.830 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:54.831 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:54.831 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:54.831 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:53:54.834 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:53:54.834 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.850+0000 7f2e0ac38640 1 Processor -- start 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.850+0000 7f2e0ac38640 1 -- start start 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.850+0000 7f2e0ac38640 10 monclient: init 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient(hunting): picked mon.noname-b con 0x7f2e040a9930 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient(hunting): picked mon.noname-c con 0x7f2e04149dc0 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0414ff40 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e040aae60 con 0x7f2e0414ff40 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e040b4310 con 0x7f2e04149dc0 2024-03-29T10:53:54.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e040afdd0 con 0x7f2e040a9930 2024-03-29T10:53:54.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:54.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.851+0000 7f2e0ac38640 10 monclient(hunting): authenticate will time out at 909.120726s 2024-03-29T10:53:54.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.858+0000 7f2e089d4640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2e040a9930 0x7f2e04143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57199/0 (socket says 172.21.15.45:57199) 2024-03-29T10:53:54.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.858+0000 7f2e089d4640 1 -- 172.21.15.45:0/1018331706 learned_addr learned my addr 172.21.15.45:0/1018331706 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364098437 0 0) 0x7f2e040aae60 con 0x7f2e0414ff40 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dec003520 con 0x7f2e0414ff40 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737934080 0 0) 0x7f2e040afdd0 con 0x7f2e040a9930 2024-03-29T10:53:54.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:53:54.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.859+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e040aae60 con 0x7f2e040a9930 2024-03-29T10:53:54.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488415807 0 0) 0x7f2dec003520 con 0x7f2e0414ff40 2024-03-29T10:53:54.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 >> v1:172.21.15.45:6790/0 conn(0x7f2e040a9930 legacy=0x7f2e04143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 >> v1:172.21.15.116:6789/0 conn(0x7f2e04149dc0 legacy=0x7f2e0414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:54.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:54.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 monclient: found mon.noname-a 2024-03-29T10:53:54.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:54.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0414a950 con 0x7f2e0414ff40 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e0ac38640 5 monclient: authenticate success, global_id 4296 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e0ac38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2df4001690 con 0x7f2e0414ff40 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: dump: 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:54.846 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e0ac38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 1 -- 172.21.15.45:0/1018331706 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2df40017f0 con 0x7f2e0414ff40 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e0ac38640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.862+0000 7f2e0ac38640 4 set_mon_vals no callback set 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.863+0000 7f2e0ac38640 10 monclient: shutdown 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.863+0000 7f2e0ac38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.863+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1018331706 >> v1:172.21.15.45:6789/0 conn(0x7f2e0414ff40 legacy=0x7f2e04152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.864+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1018331706 shutdown_connections 2024-03-29T10:53:54.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.864+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1018331706 >> 172.21.15.45:0/1018331706 conn(0x7f2e040b35e0 msgr2=0x7f2e040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:54.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.866+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1018331706 shutdown_connections 2024-03-29T10:53:54.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.866+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1018331706 wait complete. 2024-03-29T10:53:54.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 10 monclient: build_initial_monmap 2024-03-29T10:53:54.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 10 monclient: monmap: 2024-03-29T10:53:54.848 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 1 librados: starting msgr at 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 1 librados: starting objecter 2024-03-29T10:53:54.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 1 Processor -- start 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 1 -- start start 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 1 librados: setting wanted keys 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 1 librados: calling monclient init 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.867+0000 7f2e0ac38640 10 monclient: init 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient(hunting): picked mon.noname-b con 0x7f2e040a9930 addr v1:172.21.15.116:6789/0 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient(hunting): picked mon.noname-c con 0x7f2e04149dc0 addr v1:172.21.15.45:6790/0 2024-03-29T10:53:54.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0414ff40 addr v1:172.21.15.45:6789/0 2024-03-29T10:53:54.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e04147a10 con 0x7f2e0414ff40 2024-03-29T10:53:54.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e04144820 con 0x7f2e040a9930 2024-03-29T10:53:54.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e04144a90 con 0x7f2e04149dc0 2024-03-29T10:53:54.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:54.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.868+0000 7f2e0ac38640 10 monclient(hunting): authenticate will time out at 909.137844s 2024-03-29T10:53:54.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e03fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2e04149dc0 0x7f2e04147ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57303/0 (socket says 172.21.15.45:57303) 2024-03-29T10:53:54.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e03fff640 1 -- 172.21.15.45:0/1125948445 learned_addr learned my addr 172.21.15.45:0/1125948445 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 10 client.?.objecter ms_handle_connect 0x7f2e04149dc0 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608508871 0 0) 0x7f2e04144a90 con 0x7f2e04149dc0 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.875+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0004eb0 con 0x7f2e04149dc0 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480929282 0 0) 0x7f2df0004eb0 con 0x7f2e04149dc0 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e04144a90 con 0x7f2e04149dc0 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2dfc004400 con 0x7f2e04149dc0 2024-03-29T10:53:54.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): dump: 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:54.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 client.?.objecter ms_handle_connect 0x7f2e040a9930 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160350099 0 0) 0x7f2e04144820 con 0x7f2e040a9930 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.876+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0004eb0 con 0x7f2e040a9930 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.877+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270408528 0 0) 0x7f2df0004eb0 con 0x7f2e040a9930 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.877+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e04144820 con 0x7f2e040a9930 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.877+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2df80018f0 con 0x7f2e040a9930 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.877+0000 7f2e017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.877+0000 7f2e017fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.877+0000 7f2e017fa640 10 monclient(hunting): dump: 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:54.862 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:54.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 277074131 0 0) 0x7f2e04144820 con 0x7f2e040a9930 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 >> v1:172.21.15.45:6790/0 conn(0x7f2e04149dc0 legacy=0x7f2e04147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 >> v1:172.21.15.45:6789/0 conn(0x7f2e0414ff40 legacy=0x7f2e041485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 10 monclient: _finish_hunting 0 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 1 monclient: found mon.b 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e040ac5c0 con 0x7f2e040a9930 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 10 monclient: _finish_auth 0 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 10 monclient: _check_auth_tickets 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.878+0000 7f2e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:54.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 5 monclient: authenticate success, global_id 4294 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 10 monclient: _renew_subs 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e040ac810 con 0x7f2e040a9930 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 10 monclient: _renew_subs 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e041e72b0 con 0x7f2e040a9930 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e0ac38640 1 librados: init done 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2df8001d40 con 0x7f2e040a9930 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2df8004a00 con 0x7f2e040a9930 2024-03-29T10:53:54.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.879+0000 7f2e017fa640 10 monclient: dump: 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:53:54.868 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.880+0000 7f2e027fc640 4 set_mon_vals no callback set 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.881+0000 7f2e0ac38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.882+0000 7f2e0ac38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.882+0000 7f2e0ac38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.882+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de4005080 con 0x7f2e040a9930 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.882+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f2df801a9d0 con 0x7f2e040a9930 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.883+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f2df8084920 con 0x7f2e040a9930 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.883+0000 7f2e017fa640 10 client.4294.objecter ms_dispatch 0x7f2e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.883+0000 7f2e017fa640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:54.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.883+0000 7f2e017fa640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.883+0000 7f2e017fa640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T10:53:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.883+0000 7f2e03fff640 10 monclient: get_auth_request con 0x7f2df006def0 auth_method 0 2024-03-29T10:53:54.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.884+0000 7f2e017fa640 10 client.4294.objecter ms_handle_connect 0x7f2df006def0 2024-03-29T10:53:54.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.892+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2df8050200 con 0x7f2e040a9930 2024-03-29T10:53:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.892+0000 7f2e017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:54.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:54.892+0000 7f2e017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.223+0000 7f2e0ac38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.223+0000 7f2e0ac38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.223+0000 7f2e0ac38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.223+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2de4005330 con 0x7f2e040a9930 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.224+0000 7f2e017fa640 1 -- 172.21.15.45:0/1125948445 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f2df8052130 con 0x7f2e040a9930 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.224+0000 7f2e017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.224+0000 7f2e017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:55.206 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:53:55.206 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:55.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 10 librados: watch_flush enter 2024-03-29T10:53:55.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 10 librados: watch_flush exit 2024-03-29T10:53:55.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T10:53:55.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 10 client.4294.objecter successfully canceled tick 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 >> v1:172.21.15.116:6816/3290579615 conn(0x7f2df006def0 legacy=0x7f2df00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 10 monclient: shutdown 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 >> v1:172.21.15.116:6789/0 conn(0x7f2e040a9930 legacy=0x7f2e040abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 shutdown_connections 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.228+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 >> 172.21.15.45:0/1125948445 conn(0x7f2e040b35e0 msgr2=0x7f2e040b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.231+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 shutdown_connections 2024-03-29T10:53:55.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.231+0000 7f2e0ac38640 1 -- 172.21.15.45:0/1125948445 wait complete. 2024-03-29T10:53:55.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:53:55.232+0000 7f2e0ac38640 1 librados: shutdown 2024-03-29T10:53:55.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:02.039 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:02.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.214+0000 7f977a631640 10 monclient: get_monmap_and_config 2024-03-29T10:54:02.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.214+0000 7f977a631640 10 monclient: build_initial_monmap 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.214+0000 7f977a631640 10 monclient: monmap: 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:02.215242+0000 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:02.215242+0000 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.216+0000 7f977a631640 1 Processor -- start 2024-03-29T10:54:02.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.216+0000 7f977a631640 1 -- start start 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.216+0000 7f977a631640 10 monclient: init 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.217+0000 7f977a631640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 10 monclient(hunting): picked mon.noname-c con 0x7f97740a9750 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 10 monclient(hunting): picked mon.noname-a con 0x7f97740ae220 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 10 monclient(hunting): picked mon.noname-b con 0x7f977413b9d0 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:02.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774142490 con 0x7f97740ae220 2024-03-29T10:54:02.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740ac130 con 0x7f97740a9750 2024-03-29T10:54:02.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740ac410 con 0x7f977413b9d0 2024-03-29T10:54:02.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:02.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.218+0000 7f977a631640 10 monclient(hunting): authenticate will time out at 916.487230s 2024-03-29T10:54:02.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.221+0000 7f977962f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f97740a9750 0x7f9774141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:54293/0 (socket says 172.21.15.45:54293) 2024-03-29T10:54:02.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.221+0000 7f977962f640 1 -- 172.21.15.45:0/355110015 learned_addr learned my addr 172.21.15.45:0/355110015 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:02.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025790311 0 0) 0x7f97740ac410 con 0x7f977413b9d0 2024-03-29T10:54:02.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9764003520 con 0x7f977413b9d0 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635394289 0 0) 0x7f9764003520 con 0x7f977413b9d0 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.222+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 >> v1:172.21.15.116:6789/0 conn(0x7f97740a9750 legacy=0x7f9774141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 >> v1:172.21.15.45:6789/0 conn(0x7f97740ae220 legacy=0x7f977413b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 1 monclient: found mon.noname-b 2024-03-29T10:54:02.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740afa40 con 0x7f977413b9d0 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f976c001690 con 0x7f977413b9d0 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f977a631640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f977a631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f9773fff640 10 monclient: dump: 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.209 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.226+0000 7f977a631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.227+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f976c0017f0 con 0x7f977413b9d0 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.227+0000 7f9773fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.227+0000 7f9773fff640 1 -- 172.21.15.45:0/355110015 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f976c001bc0 con 0x7f977413b9d0 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.227+0000 7f9773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.227+0000 7f9773fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.227+0000 7f9773fff640 10 monclient: dump: 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:02.210 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:02.211 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.228+0000 7f977a631640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.228+0000 7f977a631640 4 set_mon_vals no callback set 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.229+0000 7f977a631640 10 monclient: shutdown 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.229+0000 7f977a631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.229+0000 7f977a631640 1 -- 172.21.15.45:0/355110015 >> v1:172.21.15.45:6790/0 conn(0x7f977413b9d0 legacy=0x7f9774149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.231+0000 7f977a631640 1 -- 172.21.15.45:0/355110015 shutdown_connections 2024-03-29T10:54:02.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.231+0000 7f977a631640 1 -- 172.21.15.45:0/355110015 >> 172.21.15.45:0/355110015 conn(0x7f97740673e0 msgr2=0x7f97740b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:02.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.232+0000 7f977a631640 1 -- 172.21.15.45:0/355110015 shutdown_connections 2024-03-29T10:54:02.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.232+0000 7f977a631640 1 -- 172.21.15.45:0/355110015 wait complete. 2024-03-29T10:54:02.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.233+0000 7f977a631640 10 monclient: build_initial_monmap 2024-03-29T10:54:02.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.233+0000 7f977a631640 10 monclient: monmap: 2024-03-29T10:54:02.214 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:02.214 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:02.215 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.234+0000 7f977a631640 1 librados: starting msgr at 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.234+0000 7f977a631640 1 librados: starting objecter 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.234+0000 7f977a631640 1 Processor -- start 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.235+0000 7f977a631640 1 -- start start 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.235+0000 7f977a631640 1 librados: setting wanted keys 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.235+0000 7f977a631640 1 librados: calling monclient init 2024-03-29T10:54:02.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.235+0000 7f977a631640 10 monclient: init 2024-03-29T10:54:02.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:02.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient(hunting): picked mon.noname-c con 0x7f97740a9750 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient(hunting): picked mon.noname-b con 0x7f97740ae220 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient(hunting): picked mon.noname-a con 0x7f977413b9d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774141520 con 0x7f977413b9d0 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774140b50 con 0x7f97740ae220 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774140f70 con 0x7f97740a9750 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:02.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.236+0000 7f977a631640 10 monclient(hunting): authenticate will time out at 916.505890s 2024-03-29T10:54:02.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.240+0000 7f9779e30640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f977413b9d0 0x7f97741dd760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60449/0 (socket says 172.21.15.45:60449) 2024-03-29T10:54:02.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.240+0000 7f9779e30640 1 -- 172.21.15.45:0/1643095899 learned_addr learned my addr 172.21.15.45:0/1643095899 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:02.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.240+0000 7f9771ffb640 10 client.?.objecter ms_handle_connect 0x7f977413b9d0 2024-03-29T10:54:02.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.240+0000 7f9771ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.244+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985879930 0 0) 0x7f9774141520 con 0x7f977413b9d0 2024-03-29T10:54:02.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.244+0000 7f9771ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.244+0000 7f9771ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.244+0000 7f9771ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.244+0000 7f9771ffb640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:54:02.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.244+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9758004eb0 con 0x7f977413b9d0 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441351044 0 0) 0x7f9758004eb0 con 0x7f977413b9d0 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9774141520 con 0x7f977413b9d0 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f976c001b00 con 0x7f977413b9d0 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:02.227 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 client.?.objecter ms_handle_connect 0x7f97740a9750 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 client.?.objecter ms_handle_connect 0x7f97740ae220 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3911407327 0 0) 0x7f9774141520 con 0x7f977413b9d0 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 >> v1:172.21.15.45:6790/0 conn(0x7f97740a9750 legacy=0x7f97741d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 >> v1:172.21.15.116:6789/0 conn(0x7f97740ae220 legacy=0x7f977413f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 monclient: found mon.a 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9774141100 con 0x7f977413b9d0 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f9771ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f977a631640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:54:02.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f977a631640 10 monclient: _renew_subs 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f977a631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.245+0000 7f977a631640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9774136b90 con 0x7f977413b9d0 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.246+0000 7f977a631640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.246+0000 7f977a631640 10 monclient: _renew_subs 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.246+0000 7f977a631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.246+0000 7f977a631640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97740b3130 con 0x7f977413b9d0 2024-03-29T10:54:02.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.246+0000 7f977a631640 1 librados: init done 2024-03-29T10:54:02.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.247+0000 7f975f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.247+0000 7f975f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.247+0000 7f975f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.247+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9754005080 con 0x7f977413b9d0 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.247+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f976c001c60 con 0x7f977413b9d0 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.250+0000 7f9771ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.250+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f976c003a40 con 0x7f977413b9d0 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.250+0000 7f9771ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.250+0000 7f9771ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.250+0000 7f9771ffb640 10 monclient: dump: 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:02.233 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.250+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f976c003c60 con 0x7f977413b9d0 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.251+0000 7f977962f640 10 monclient: get_auth_request con 0x7f9758071fe0 auth_method 0 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.252+0000 7f97737fe640 4 set_mon_vals no callback set 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.251+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f976c084b60 con 0x7f977413b9d0 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.252+0000 7f9771ffb640 10 client.4302.objecter ms_dispatch 0x7f9774000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.252+0000 7f9771ffb640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.252+0000 7f9771ffb640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.252+0000 7f9771ffb640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T10:54:02.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.252+0000 7f9771ffb640 10 client.4302.objecter ms_handle_connect 0x7f9758071fe0 2024-03-29T10:54:02.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.260+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f976c0504f0 con 0x7f977413b9d0 2024-03-29T10:54:02.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.260+0000 7f9771ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.260+0000 7f9771ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:02.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.587+0000 7f975f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:02.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.587+0000 7f975f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:02.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.587+0000 7f975f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:02.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.587+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9754005330 con 0x7f977413b9d0 2024-03-29T10:54:02.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.591+0000 7f9771ffb640 1 -- 172.21.15.45:0/1643095899 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f976c052420 con 0x7f977413b9d0 2024-03-29T10:54:02.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.591+0000 7f9771ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:02.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.591+0000 7f9771ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:02.573 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:02.573 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:02.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.595+0000 7f975f7fe640 10 librados: watch_flush enter 2024-03-29T10:54:02.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 10 librados: watch_flush exit 2024-03-29T10:54:02.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T10:54:02.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 10 client.4302.objecter successfully canceled tick 2024-03-29T10:54:02.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 >> v1:172.21.15.116:6816/3290579615 conn(0x7f9758071fe0 legacy=0x7f97580744a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 10 monclient: shutdown 2024-03-29T10:54:02.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:02.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 >> v1:172.21.15.45:6789/0 conn(0x7f977413b9d0 legacy=0x7f97741dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 shutdown_connections 2024-03-29T10:54:02.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.598+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 >> 172.21.15.45:0/1643095899 conn(0x7f97740673e0 msgr2=0x7f9774069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:02.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.599+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 shutdown_connections 2024-03-29T10:54:02.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.602+0000 7f975f7fe640 1 -- 172.21.15.45:0/1643095899 wait complete. 2024-03-29T10:54:02.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:02.602+0000 7f975f7fe640 1 librados: shutdown 2024-03-29T10:54:02.601 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:09.409 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.585+0000 7f0799ff4640 10 monclient: get_monmap_and_config 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.585+0000 7f0799ff4640 10 monclient: build_initial_monmap 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.585+0000 7f0799ff4640 10 monclient: monmap: 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:09.586449+0000 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:09.586449+0000 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:09.568 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:09.569 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:09.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.588+0000 7f0799ff4640 1 Processor -- start 2024-03-29T10:54:09.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.588+0000 7f0799ff4640 1 -- start start 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.588+0000 7f0799ff4640 10 monclient: init 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f07940a9ac0 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f079413ee90 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f07940af900 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:09.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940afd10 con 0x7f07940af900 2024-03-29T10:54:09.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940af6f0 con 0x7f079413ee90 2024-03-29T10:54:09.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940aac80 con 0x7f07940a9ac0 2024-03-29T10:54:09.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:09.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0799ff4640 10 monclient(hunting): authenticate will time out at 923.858753s 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0793fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f07940af900 0x7f0794149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59416/0 (socket says 172.21.15.45:59416) 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f0793fff640 1 -- 172.21.15.45:0/2267244751 learned_addr learned my addr 172.21.15.45:0/2267244751 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.589+0000 7f07937fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07940a9ac0 0x7f07940adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38306/0 (socket says 172.21.15.45:38306) 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943644103 0 0) 0x7f07940afd10 con 0x7f07940af900 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:54:09.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f077c003520 con 0x7f07940af900 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125884451 0 0) 0x7f07940aac80 con 0x7f07940a9ac0 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07940afd10 con 0x7f07940a9ac0 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874680828 0 0) 0x7f07940af6f0 con 0x7f079413ee90 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07940aac80 con 0x7f079413ee90 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068005528 0 0) 0x7f077c003520 con 0x7f07940af900 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 >> v1:172.21.15.45:6790/0 conn(0x7f07940a9ac0 legacy=0x7f07940adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 >> v1:172.21.15.116:6789/0 conn(0x7f079413ee90 legacy=0x7f079413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 monclient: found mon.noname-a 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07940aaf60 con 0x7f07940af900 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:09.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f078c001a10 con 0x7f07940af900 2024-03-29T10:54:09.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f07927fc640 10 monclient: dump: 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.583 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:09.584 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:09.585 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:09.585 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:09.585 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:09.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f0799ff4640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:54:09.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.590+0000 7f0799ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:09.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f078c001c60 con 0x7f07940af900 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f07927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f07927fc640 1 -- 172.21.15.45:0/2267244751 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f078c001ff0 con 0x7f07940af900 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f07927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f07927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f07927fc640 10 monclient: dump: 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:09.586 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f0799ff4640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.591+0000 7f0799ff4640 4 set_mon_vals no callback set 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 10 monclient: shutdown 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 1 -- 172.21.15.45:0/2267244751 >> v1:172.21.15.45:6789/0 conn(0x7f07940af900 legacy=0x7f0794149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 1 -- 172.21.15.45:0/2267244751 shutdown_connections 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 1 -- 172.21.15.45:0/2267244751 >> 172.21.15.45:0/2267244751 conn(0x7f07940b35e0 msgr2=0x7f07940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 1 -- 172.21.15.45:0/2267244751 shutdown_connections 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.592+0000 7f0799ff4640 1 -- 172.21.15.45:0/2267244751 wait complete. 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.593+0000 7f0799ff4640 10 monclient: build_initial_monmap 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.593+0000 7f0799ff4640 10 monclient: monmap: 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:09.587 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 1 librados: starting msgr at 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 1 librados: starting objecter 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 1 Processor -- start 2024-03-29T10:54:09.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 1 -- start start 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 1 librados: setting wanted keys 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 1 librados: calling monclient init 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.594+0000 7f0799ff4640 10 monclient: init 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0799ff4640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0799ff4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0799ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f07940a9ac0 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0799ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f07940af900 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0799ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f079413ee90 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0799ff4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0794141c70 con 0x7f07940af900 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0793fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f079413ee90 0x7f07941de640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39274/0 (socket says 172.21.15.45:39274) 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.595+0000 7f0793fff640 1 -- 172.21.15.45:0/2234931222 learned_addr learned my addr 172.21.15.45:0/2234931222 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.596+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0794141850 con 0x7f07940a9ac0 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.596+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079413c7d0 con 0x7f079413ee90 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.596+0000 7f0799ff4640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:09.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.596+0000 7f0799ff4640 10 monclient(hunting): authenticate will time out at 923.865358s 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 client.?.objecter ms_handle_connect 0x7f079413ee90 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723643919 0 0) 0x7f079413c7d0 con 0x7f079413ee90 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0780004eb0 con 0x7f079413ee90 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.598+0000 7f077bfff640 10 client.?.objecter ms_handle_connect 0x7f07940a9ac0 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165323992 0 0) 0x7f0780004eb0 con 0x7f079413ee90 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f079413c7d0 con 0x7f079413ee90 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f078c001f70 con 0x7f079413ee90 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:09.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508832022 0 0) 0x7f079413c7d0 con 0x7f079413ee90 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 >> v1:172.21.15.116:6789/0 conn(0x7f07940a9ac0 legacy=0x7f079413c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.599+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 >> v1:172.21.15.45:6789/0 conn(0x7f07940af900 legacy=0x7f0794136750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 1 monclient: found mon.c 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079413e2e0 con 0x7f079413ee90 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 10 monclient: _finish_auth 0 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f077bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f0799ff4640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f0799ff4640 10 monclient: _renew_subs 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f0799ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:09.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.600+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0794142380 con 0x7f079413ee90 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.601+0000 7f0799ff4640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.601+0000 7f0799ff4640 10 monclient: _renew_subs 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.601+0000 7f0799ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.601+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0794137040 con 0x7f079413ee90 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.601+0000 7f0799ff4640 1 librados: init done 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.602+0000 7f0799ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.602+0000 7f0799ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:09.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.602+0000 7f0799ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.602+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079406dfd0 con 0x7f079413ee90 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f078c001560 con 0x7f079413ee90 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f077bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f078c005110 con 0x7f079413ee90 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f077bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f077bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f077bfff640 10 monclient: dump: 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:09.595 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.605+0000 7f0791ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.608+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f078c01b0e0 con 0x7f079413ee90 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.609+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f078c084f20 con 0x7f079413ee90 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.609+0000 7f077bfff640 10 client.4313.objecter ms_dispatch 0x7f0794000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.609+0000 7f077bfff640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:09.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.609+0000 7f077bfff640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:09.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.609+0000 7f077bfff640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T10:54:09.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.609+0000 7f07937fe640 10 monclient: get_auth_request con 0x7f078006dcc0 auth_method 0 2024-03-29T10:54:09.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.610+0000 7f077bfff640 10 client.4313.objecter ms_handle_connect 0x7f078006dcc0 2024-03-29T10:54:09.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.614+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f078c050800 con 0x7f079413ee90 2024-03-29T10:54:09.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.615+0000 7f077bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:09.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.615+0000 7f077bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:09.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.969+0000 7f0799ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:09.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.969+0000 7f0799ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:09.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.969+0000 7f0799ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:09.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.969+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0794075f10 con 0x7f079413ee90 2024-03-29T10:54:09.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.974+0000 7f077bfff640 1 -- 172.21.15.45:0/2234931222 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f078c052730 con 0x7f079413ee90 2024-03-29T10:54:09.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.974+0000 7f077bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:09.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.974+0000 7f077bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:09.955 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:09.955 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:09.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.977+0000 7f0799ff4640 10 librados: watch_flush enter 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 10 librados: watch_flush exit 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 10 client.4313.objecter successfully canceled tick 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 >> v1:172.21.15.116:6816/3290579615 conn(0x7f078006dcc0 legacy=0x7f0780070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 10 monclient: shutdown 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:09.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.980+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 >> v1:172.21.15.45:6790/0 conn(0x7f079413ee90 legacy=0x7f07941de640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.982+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 shutdown_connections 2024-03-29T10:54:09.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.982+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 >> 172.21.15.45:0/2234931222 conn(0x7f07940b35e0 msgr2=0x7f07940adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:09.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.983+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 shutdown_connections 2024-03-29T10:54:09.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.984+0000 7f0799ff4640 1 -- 172.21.15.45:0/2234931222 wait complete. 2024-03-29T10:54:09.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:09.984+0000 7f0799ff4640 1 librados: shutdown 2024-03-29T10:54:09.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:16.791 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:16.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.969+0000 7f3e8e115640 10 monclient: get_monmap_and_config 2024-03-29T10:54:16.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.969+0000 7f3e8e115640 10 monclient: build_initial_monmap 2024-03-29T10:54:16.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.969+0000 7f3e8e115640 10 monclient: monmap: 2024-03-29T10:54:16.951 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:16.951 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:16.971140+0000 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:16.971140+0000 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:16.952 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:16.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.970+0000 7f3e8e115640 1 Processor -- start 2024-03-29T10:54:16.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.970+0000 7f3e8e115640 1 -- start start 2024-03-29T10:54:16.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.970+0000 7f3e8e115640 10 monclient: init 2024-03-29T10:54:16.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.971+0000 7f3e8e115640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:16.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.971+0000 7f3e8e115640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:16.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.971+0000 7f3e8e115640 10 monclient(hunting): picked mon.noname-c con 0x7f3e880b0860 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:16.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.974+0000 7f3e86ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3e8813ef30 0x7f3e880af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54275/0 (socket says 172.21.15.45:54275) 2024-03-29T10:54:16.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.974+0000 7f3e86ffd640 1 -- 172.21.15.45:0/1038754168 learned_addr learned my addr 172.21.15.45:0/1038754168 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:16.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.974+0000 7f3e8e115640 10 monclient(hunting): picked mon.noname-a con 0x7f3e8813ef30 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:16.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.974+0000 7f3e8e115640 10 monclient(hunting): picked mon.noname-b con 0x7f3e8806d9d0 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:16.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.977+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880b0c40 con 0x7f3e8813ef30 2024-03-29T10:54:16.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.977+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880aed10 con 0x7f3e880b0860 2024-03-29T10:54:16.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.977+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880ab1b0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.977+0000 7f3e8e115640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:16.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.977+0000 7f3e8e115640 10 monclient(hunting): authenticate will time out at 931.247529s 2024-03-29T10:54:16.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678268509 0 0) 0x7f3e880aed10 con 0x7f3e880b0860 2024-03-29T10:54:16.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e70005730 con 0x7f3e880b0860 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188237082 0 0) 0x7f3e880b0c40 con 0x7f3e8813ef30 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e880aed10 con 0x7f3e8813ef30 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 957228012 0 0) 0x7f3e880aed10 con 0x7f3e8813ef30 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 >> v1:172.21.15.45:6790/0 conn(0x7f3e8806d9d0 legacy=0x7f3e8814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 >> v1:172.21.15.116:6789/0 conn(0x7f3e880b0860 legacy=0x7f3e8813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:16.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 monclient: found mon.noname-a 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880ab490 con 0x7f3e8813ef30 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e7c000ed0 con 0x7f3e8813ef30 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:16.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.978+0000 7f3e867fc640 10 monclient: dump: 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 5 monclient: authenticate success, global_id 4308 2024-03-29T10:54:16.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:16.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e7c0010f0 con 0x7f3e8813ef30 2024-03-29T10:54:16.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:16.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e867fc640 1 -- 172.21.15.45:0/1038754168 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e7c001400 con 0x7f3e8813ef30 2024-03-29T10:54:16.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:16.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e867fc640 10 monclient: dump: 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 4 set_mon_vals no callback set 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 10 monclient: shutdown 2024-03-29T10:54:16.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.979+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 >> v1:172.21.15.45:6789/0 conn(0x7f3e8813ef30 legacy=0x7f3e880af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.981+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 shutdown_connections 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.981+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 >> 172.21.15.45:0/1038754168 conn(0x7f3e8806b660 msgr2=0x7f3e8806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.981+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 shutdown_connections 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.981+0000 7f3e8e115640 1 -- 172.21.15.45:0/1038754168 wait complete. 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 10 monclient: build_initial_monmap 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 10 monclient: monmap: 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:16.966 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:16.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 1 librados: starting msgr at 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 1 librados: starting objecter 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 1 Processor -- start 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 1 -- start start 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 1 librados: setting wanted keys 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 1 librados: calling monclient init 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.982+0000 7f3e8e115640 10 monclient: init 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.983+0000 7f3e8e115640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.983+0000 7f3e8e115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:16.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.984+0000 7f3e8e115640 10 monclient(hunting): picked mon.noname-a con 0x7f3e8806d9d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e8e115640 10 monclient(hunting): picked mon.noname-c con 0x7f3e880b0860 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e8e115640 10 monclient(hunting): picked mon.noname-b con 0x7f3e8813ef30 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e8e115640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e881d3490 con 0x7f3e8806d9d0 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e8e115640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880b2a80 con 0x7f3e8813ef30 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e8e115640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880b3e10 con 0x7f3e880b0860 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e86ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f3e880b0860 0x7f3e881d3920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35095/0 (socket says 172.21.15.45:35095) 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e8e115640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.987+0000 7f3e86ffd640 1 -- 172.21.15.45:0/2704630556 learned_addr learned my addr 172.21.15.45:0/2704630556 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:16.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.988+0000 7f3e8e115640 10 monclient(hunting): authenticate will time out at 931.257998s 2024-03-29T10:54:16.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.988+0000 7f3e6ffff640 10 client.?.objecter ms_handle_connect 0x7f3e880b0860 2024-03-29T10:54:16.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:16.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 client.?.objecter ms_handle_connect 0x7f3e8806d9d0 2024-03-29T10:54:16.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:16.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544729313 0 0) 0x7f3e881d3490 con 0x7f3e8806d9d0 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e740092c0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095446414 0 0) 0x7f3e880b3e10 con 0x7f3e880b0860 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.991+0000 7f3e6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e881d3490 con 0x7f3e880b0860 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 client.?.objecter ms_handle_connect 0x7f3e8813ef30 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106850587 0 0) 0x7f3e880b2a80 con 0x7f3e8813ef30 2024-03-29T10:54:16.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:54:16.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.992+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e880b3e10 con 0x7f3e8813ef30 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2032755190 0 0) 0x7f3e740092c0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e74009a30 con 0x7f3e8806d9d0 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e78001950 con 0x7f3e8806d9d0 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient(hunting): dump: 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:16.977 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 786559650 0 0) 0x7f3e74009a30 con 0x7f3e8806d9d0 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 >> v1:172.21.15.45:6790/0 conn(0x7f3e880b0860 legacy=0x7f3e881d3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 >> v1:172.21.15.116:6789/0 conn(0x7f3e8813ef30 legacy=0x7f3e881dde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 monclient: found mon.a 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e88139e40 con 0x7f3e8806d9d0 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: _finish_auth 0 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:16.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e78003610 con 0x7f3e8806d9d0 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3e780039e0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e6ffff640 10 monclient: dump: 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:16.979 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:16.980 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:16.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e8e115640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:54:16.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e8e115640 10 monclient: _renew_subs 2024-03-29T10:54:16.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e8e115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:16.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e881d4030 con 0x7f3e8806d9d0 2024-03-29T10:54:16.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.993+0000 7f3e85ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.995+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f3e78003b40 con 0x7f3e8806d9d0 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e8e115640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e8e115640 10 monclient: _renew_subs 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e8e115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e880b2e70 con 0x7f3e8806d9d0 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e86ffd640 10 monclient: get_auth_request con 0x7f3e74072450 auth_method 0 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e8e115640 1 librados: init done 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e6ffff640 10 client.4311.objecter ms_handle_connect 0x7f3e74072450 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f3e780536d0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e6ffff640 10 client.4311.objecter ms_dispatch 0x7f3e88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e6ffff640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e6ffff640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:16.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.996+0000 7f3e6ffff640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T10:54:16.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.998+0000 7f3e8e115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:16.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.998+0000 7f3e8e115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:16.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.998+0000 7f3e8e115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:16.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:16.998+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e880b30c0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.009+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e7804f1e0 con 0x7f3e8806d9d0 2024-03-29T10:54:16.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.009+0000 7f3e6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:16.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.009+0000 7f3e6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.339+0000 7f3e8e115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.339+0000 7f3e8e115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.339+0000 7f3e8e115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.339+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e88075f10 con 0x7f3e8806d9d0 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.340+0000 7f3e6ffff640 1 -- 172.21.15.45:0/2704630556 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f3e78051110 con 0x7f3e8806d9d0 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.340+0000 7f3e6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:17.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.340+0000 7f3e6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:17.324 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:17.324 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:17.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.345+0000 7f3e8e115640 10 librados: watch_flush enter 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 10 librados: watch_flush exit 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 10 client.4311.objecter successfully canceled tick 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 >> v1:172.21.15.116:6816/3290579615 conn(0x7f3e74072450 legacy=0x7f3e74074910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 10 monclient: shutdown 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 >> v1:172.21.15.45:6789/0 conn(0x7f3e8806d9d0 legacy=0x7f3e880b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 shutdown_connections 2024-03-29T10:54:17.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.346+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 >> 172.21.15.45:0/2704630556 conn(0x7f3e8806b660 msgr2=0x7f3e880ac8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:17.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.347+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 shutdown_connections 2024-03-29T10:54:17.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.347+0000 7f3e8e115640 1 -- 172.21.15.45:0/2704630556 wait complete. 2024-03-29T10:54:17.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:17.347+0000 7f3e8e115640 1 librados: shutdown 2024-03-29T10:54:17.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:24.156 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.321+0000 7fead4ea8640 10 monclient: get_monmap_and_config 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.321+0000 7fead4ea8640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.321+0000 7fead4ea8640 10 monclient: monmap: 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:24.323092+0000 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:24.323092+0000 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.303 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.322+0000 7fead4ea8640 1 Processor -- start 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.322+0000 7fead4ea8640 1 -- start start 2024-03-29T10:54:24.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.322+0000 7fead4ea8640 10 monclient: init 2024-03-29T10:54:24.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:24.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fead006ee40 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:24.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fead00aeda0 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:24.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fead013c470 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead006fdb0 con 0x7fead006ee40 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead0071090 con 0x7fead00aeda0 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead0071370 con 0x7fead013c470 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7fead4ea8640 10 monclient(hunting): authenticate will time out at 938.593450s 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7feace59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fead006ee40 0x7fead013bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50648/0 (socket says 172.21.15.45:50648) 2024-03-29T10:54:24.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.323+0000 7feace59c640 1 -- 172.21.15.45:0/3478493693 learned_addr learned my addr 172.21.15.45:0/3478493693 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:24.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feaced9d640 1 -- 172.21.15.45:0/3478493693 >> v1:172.21.15.45:6790/0 conn(0x7fead013c470 legacy=0x7fead014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T10:54:24.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feaced9d640 1 -- 172.21.15.45:0/3478493693 >> v1:172.21.15.45:6790/0 conn(0x7fead013c470 legacy=0x7fead014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T10:54:24.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feaced9d640 1 --1- 172.21.15.45:0/3478493693 >> v1:172.21.15.45:6790/0 conn(0x7fead013c470 0x7fead014a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T10:54:24.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541347854 0 0) 0x7fead0071090 con 0x7fead00aeda0 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab8005730 con 0x7fead00aeda0 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940770654 0 0) 0x7fead006fdb0 con 0x7fead006ee40 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead0071090 con 0x7fead006ee40 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.324+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2904041694 0 0) 0x7feab8005730 con 0x7fead00aeda0 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 >> v1:172.21.15.45:6790/0 conn(0x7fead013c470 legacy=0x7fead014a640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 >> v1:172.21.15.45:6789/0 conn(0x7fead006ee40 legacy=0x7fead013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 monclient: found mon.noname-c 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0070220 con 0x7fead00aeda0 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7feac4001690 con 0x7fead00aeda0 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: dump: 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.312 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7fead4ea8640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7fead4ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac40017f0 con 0x7fead00aeda0 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7fead4ea8640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7fead4ea8640 4 set_mon_vals no callback set 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 1 -- 172.21.15.45:0/3478493693 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7feac4001bc0 con 0x7fead00aeda0 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.325+0000 7feacd59a640 10 monclient: dump: 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.314 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 10 monclient: shutdown 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 1 -- 172.21.15.45:0/3478493693 >> v1:172.21.15.116:6789/0 conn(0x7fead00aeda0 legacy=0x7fead0071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 1 -- 172.21.15.45:0/3478493693 shutdown_connections 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 1 -- 172.21.15.45:0/3478493693 >> 172.21.15.45:0/3478493693 conn(0x7fead0067470 msgr2=0x7fead00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 1 -- 172.21.15.45:0/3478493693 shutdown_connections 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.326+0000 7fead4ea8640 1 -- 172.21.15.45:0/3478493693 wait complete. 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.327+0000 7fead4ea8640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.327+0000 7fead4ea8640 10 monclient: monmap: 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:24.315 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 1 librados: starting msgr at 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 1 librados: starting objecter 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 1 Processor -- start 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 1 -- start start 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 1 librados: setting wanted keys 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 1 librados: calling monclient init 2024-03-29T10:54:24.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.328+0000 7fead4ea8640 10 monclient: init 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fead006ee40 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7feace59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fead006ee40 0x7fead01d5d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35352/0 (socket says 172.21.15.45:35352) 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7feace59c640 1 -- 172.21.15.45:0/394612397 learned_addr learned my addr 172.21.15.45:0/394612397 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fead013c470 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fead0141010 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead00a9860 con 0x7fead013c470 2024-03-29T10:54:24.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead0141fb0 con 0x7fead0141010 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.329+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead01424d0 con 0x7fead006ee40 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7fead4ea8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7fead4ea8640 10 monclient(hunting): authenticate will time out at 938.600007s 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 client.?.objecter ms_handle_connect 0x7fead013c470 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 client.?.objecter ms_handle_connect 0x7fead006ee40 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 client.?.objecter ms_handle_connect 0x7fead0141010 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903033527 0 0) 0x7fead00a9860 con 0x7fead013c470 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0004eb0 con 0x7fead013c470 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711095614 0 0) 0x7fead0141fb0 con 0x7fead0141010 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead00a9860 con 0x7fead0141010 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284535163 0 0) 0x7fead01424d0 con 0x7fead006ee40 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead0141fb0 con 0x7fead006ee40 2024-03-29T10:54:24.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1022516590 0 0) 0x7feab0004eb0 con 0x7fead013c470 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead01424d0 con 0x7fead013c470 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987617214 0 0) 0x7fead00a9860 con 0x7fead0141010 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feab0004eb0 con 0x7fead0141010 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7feac4001bc0 con 0x7fead013c470 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.331+0000 7feabeffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:24.325 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7feac8001ad0 con 0x7fead0141010 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:24.326 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3189247170 0 0) 0x7feab0004eb0 con 0x7fead0141010 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 >> v1:172.21.15.45:6790/0 conn(0x7fead006ee40 legacy=0x7fead01d5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 >> v1:172.21.15.45:6789/0 conn(0x7fead013c470 legacy=0x7fead00acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 1 monclient: found mon.b 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.332+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0142660 con 0x7fead0141010 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7feabeffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7feabeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7feabeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:54:24.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 10 monclient: _renew_subs 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead01d6480 con 0x7fead0141010 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 10 monclient: _renew_subs 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fead01d6780 con 0x7fead0141010 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7fead4ea8640 1 librados: init done 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac8001c30 con 0x7fead0141010 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7feabeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.333+0000 7feaccd99640 4 set_mon_vals no callback set 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.334+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7feac8004cb0 con 0x7fead0141010 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.334+0000 7fead4ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.334+0000 7fead4ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.334+0000 7fead4ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:24.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.334+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab4005080 con 0x7fead0141010 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.335+0000 7feabeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.335+0000 7feabeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.335+0000 7feabeffd640 10 monclient: dump: 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.336+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7feac801ac80 con 0x7fead0141010 2024-03-29T10:54:24.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feace59c640 10 monclient: get_auth_request con 0x7feab006df50 auth_method 0 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7feac8084bb0 con 0x7fead0141010 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feabeffd640 10 client.4318.objecter ms_dispatch 0x7fead0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feabeffd640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feabeffd640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feabeffd640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.337+0000 7feabeffd640 10 client.4318.objecter ms_handle_connect 0x7feab006df50 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.344+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feac8050490 con 0x7fead0141010 2024-03-29T10:54:24.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.344+0000 7feabeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.344+0000 7feabeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.727+0000 7fead4ea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.728+0000 7fead4ea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.728+0000 7fead4ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:24.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.728+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feab4005330 con 0x7fead0141010 2024-03-29T10:54:24.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.729+0000 7feabeffd640 1 -- 172.21.15.45:0/394612397 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7feac80523c0 con 0x7fead0141010 2024-03-29T10:54:24.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.729+0000 7feabeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.729+0000 7feabeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.710 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:24.711 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:24.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 10 librados: watch_flush enter 2024-03-29T10:54:24.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 10 librados: watch_flush exit 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 10 client.4318.objecter successfully canceled tick 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 >> v1:172.21.15.116:6816/3290579615 conn(0x7feab006df50 legacy=0x7feab0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 10 monclient: shutdown 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 >> v1:172.21.15.116:6789/0 conn(0x7fead0141010 legacy=0x7fead01de110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 shutdown_connections 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 >> 172.21.15.45:0/394612397 conn(0x7fead0067470 msgr2=0x7fead013bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 shutdown_connections 2024-03-29T10:54:24.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 -- 172.21.15.45:0/394612397 wait complete. 2024-03-29T10:54:24.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:24.733+0000 7fead4ea8640 1 librados: shutdown 2024-03-29T10:54:24.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:31.536 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.702+0000 7f2b040a8640 10 monclient: get_monmap_and_config 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.702+0000 7f2b040a8640 10 monclient: build_initial_monmap 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.702+0000 7f2b040a8640 10 monclient: monmap: 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:31.703505+0000 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:31.703505+0000 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:31.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.703+0000 7f2b040a8640 1 Processor -- start 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.703+0000 7f2b040a8640 1 -- start start 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.703+0000 7f2b040a8640 10 monclient: init 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2afc0a9750 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2afc149ac0 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2afc14fe50 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0aa4a0 con 0x7f2afc14fe50 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc14a680 con 0x7f2afc149ac0 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0af290 con 0x7f2afc0a9750 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.704+0000 7f2b040a8640 10 monclient(hunting): authenticate will time out at 945.973694s 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.706+0000 7f2b01643640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2afc149ac0 0x7f2afc149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:49370/0 (socket says 172.21.15.45:49370) 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.706+0000 7f2b01643640 1 -- 172.21.15.45:0/2459538760 learned_addr learned my addr 172.21.15.45:0/2459538760 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085156962 0 0) 0x7f2afc14a680 con 0x7f2afc149ac0 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae8003520 con 0x7f2afc149ac0 2024-03-29T10:54:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300139788 0 0) 0x7f2ae8003520 con 0x7f2afc149ac0 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 >> v1:172.21.15.45:6790/0 conn(0x7f2afc0a9750 legacy=0x7f2afc1436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.707+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 >> v1:172.21.15.45:6789/0 conn(0x7f2afc14fe50 legacy=0x7f2afc152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: _finish_hunting 0 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 1 monclient: found mon.noname-c 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc0afb40 con 0x7f2afc149ac0 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: _finish_auth 0 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: _check_auth_tickets 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2aec001690 con 0x7f2afc149ac0 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:31.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: dump: 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2aec0017f0 con 0x7f2afc149ac0 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 1 -- 172.21.15.45:0/2459538760 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2aec001c40 con 0x7f2afc149ac0 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b00e42640 10 monclient: dump: 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b040a8640 5 monclient: authenticate success, global_id 4321 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b040a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.708+0000 7f2b040a8640 4 set_mon_vals no callback set 2024-03-29T10:54:31.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.709+0000 7f2b040a8640 10 monclient: shutdown 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.709+0000 7f2b040a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.709+0000 7f2b040a8640 1 -- 172.21.15.45:0/2459538760 >> v1:172.21.15.116:6789/0 conn(0x7f2afc149ac0 legacy=0x7f2afc149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.711+0000 7f2b040a8640 1 -- 172.21.15.45:0/2459538760 shutdown_connections 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.711+0000 7f2b040a8640 1 -- 172.21.15.45:0/2459538760 >> 172.21.15.45:0/2459538760 conn(0x7f2afc0b35e0 msgr2=0x7f2afc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.712+0000 7f2b040a8640 1 -- 172.21.15.45:0/2459538760 shutdown_connections 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.712+0000 7f2b040a8640 1 -- 172.21.15.45:0/2459538760 wait complete. 2024-03-29T10:54:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.713+0000 7f2b040a8640 10 monclient: build_initial_monmap 2024-03-29T10:54:31.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.713+0000 7f2b040a8640 10 monclient: monmap: 2024-03-29T10:54:31.696 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:31.696 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:31.696 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:31.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.713+0000 7f2b040a8640 1 librados: starting msgr at 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.713+0000 7f2b040a8640 1 librados: starting objecter 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.713+0000 7f2b040a8640 1 Processor -- start 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.714+0000 7f2b040a8640 1 -- start start 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.714+0000 7f2b040a8640 1 librados: setting wanted keys 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.714+0000 7f2b040a8640 1 librados: calling monclient init 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.714+0000 7f2b040a8640 10 monclient: init 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2afc0a9750 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2afc149ac0 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2afc14fe50 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0ac050 con 0x7f2afc0a9750 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc138d00 con 0x7f2afc14fe50 2024-03-29T10:54:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0ab240 con 0x7f2afc149ac0 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.715+0000 7f2b040a8640 10 monclient(hunting): authenticate will time out at 945.984354s 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.716+0000 7f2b02645640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2afc14fe50 0x7f2afc1385f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:49484/0 (socket says 172.21.15.45:49484) 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.716+0000 7f2b02645640 1 -- 172.21.15.45:0/4009121480 learned_addr learned my addr 172.21.15.45:0/4009121480 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.716+0000 7f2af27fc640 10 client.?.objecter ms_handle_connect 0x7f2afc14fe50 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.716+0000 7f2af27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097305100 0 0) 0x7f2afc138d00 con 0x7f2afc14fe50 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:54:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0004eb0 con 0x7f2afc14fe50 2024-03-29T10:54:31.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379423541 0 0) 0x7f2ae0004eb0 con 0x7f2afc14fe50 2024-03-29T10:54:31.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2afc138d00 con 0x7f2afc14fe50 2024-03-29T10:54:31.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2af4001d30 con 0x7f2afc14fe50 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.717+0000 7f2af27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.718+0000 7f2af27fc640 10 monclient(hunting): dump: 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.718+0000 7f2af27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:31.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.718+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 578252622 0 0) 0x7f2afc138d00 con 0x7f2afc14fe50 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.718+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 >> v1:172.21.15.45:6790/0 conn(0x7f2afc149ac0 legacy=0x7f2afc13a440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 >> v1:172.21.15.45:6789/0 conn(0x7f2afc0a9750 legacy=0x7f2afc137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 1 monclient: found mon.b 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc0ab3d0 con 0x7f2afc14fe50 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2af27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:54:31.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 10 monclient: _renew_subs 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc1467a0 con 0x7f2afc14fe50 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 10 monclient: _renew_subs 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc0aca50 con 0x7f2afc14fe50 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.719+0000 7f2b040a8640 1 librados: init done 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.720+0000 7f2b040a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.720+0000 7f2b040a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2b040a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc0ac500 con 0x7f2afc14fe50 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af4002030 con 0x7f2afc14fe50 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2af27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:31.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2af4003de0 con 0x7f2afc14fe50 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2af27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2af27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.721+0000 7f2af27fc640 10 monclient: dump: 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.722+0000 7f2af3fff640 4 set_mon_vals no callback set 2024-03-29T10:54:31.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.722+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f2af4004000 con 0x7f2afc14fe50 2024-03-29T10:54:31.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.723+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f2af4084ce0 con 0x7f2afc14fe50 2024-03-29T10:54:31.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.723+0000 7f2af27fc640 10 client.4324.objecter ms_dispatch 0x7f2afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:31.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.723+0000 7f2af27fc640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:31.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.723+0000 7f2af27fc640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:31.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.723+0000 7f2b01e44640 10 monclient: get_auth_request con 0x7f2ae006db30 auth_method 0 2024-03-29T10:54:31.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.723+0000 7f2af27fc640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T10:54:31.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.726+0000 7f2af27fc640 10 client.4324.objecter ms_handle_connect 0x7f2ae006db30 2024-03-29T10:54:31.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.729+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2af40505c0 con 0x7f2afc14fe50 2024-03-29T10:54:31.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.729+0000 7f2af27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:31.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:31.729+0000 7f2af27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:32.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.051+0000 7f2b040a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.051+0000 7f2b040a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.051+0000 7f2b040a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:32.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.051+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2afc0ac500 con 0x7f2afc14fe50 2024-03-29T10:54:32.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.052+0000 7f2af27fc640 1 -- 172.21.15.45:0/4009121480 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f2afc0ac500 con 0x7f2afc14fe50 2024-03-29T10:54:32.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.052+0000 7f2af27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.052+0000 7f2af27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:32.034 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:32.034 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 10 librados: watch_flush enter 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 10 librados: watch_flush exit 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 10 client.4324.objecter successfully canceled tick 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 >> v1:172.21.15.116:6816/3290579615 conn(0x7f2ae006db30 legacy=0x7f2ae006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 10 monclient: shutdown 2024-03-29T10:54:32.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:32.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.056+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 >> v1:172.21.15.116:6789/0 conn(0x7f2afc14fe50 legacy=0x7f2afc1385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.057+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 shutdown_connections 2024-03-29T10:54:32.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.057+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 >> 172.21.15.45:0/4009121480 conn(0x7f2afc0b35e0 msgr2=0x7f2afc0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:32.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.058+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 shutdown_connections 2024-03-29T10:54:32.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.058+0000 7f2b040a8640 1 -- 172.21.15.45:0/4009121480 wait complete. 2024-03-29T10:54:32.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:32.058+0000 7f2b040a8640 1 librados: shutdown 2024-03-29T10:54:32.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:38.862 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:39.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.032+0000 7fd7dc75d640 10 monclient: get_monmap_and_config 2024-03-29T10:54:39.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.032+0000 7fd7dc75d640 10 monclient: build_initial_monmap 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.032+0000 7fd7dc75d640 10 monclient: monmap: 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:39.033761+0000 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:39.033761+0000 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:39.014 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:39.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.033+0000 7fd7dc75d640 1 Processor -- start 2024-03-29T10:54:39.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.033+0000 7fd7dc75d640 1 -- start start 2024-03-29T10:54:39.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.033+0000 7fd7dc75d640 10 monclient: init 2024-03-29T10:54:39.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7d40b0770 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7d414a700 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7d40ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d40aba70 con 0x7fd7d40b0770 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d40aecb0 con 0x7fd7d414a700 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d40af150 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:39.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.034+0000 7fd7dc75d640 10 monclient(hunting): authenticate will time out at 953.303970s 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.035+0000 7fd7d9cf8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd7d414a700 0x7fd7d40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:60116/0 (socket says 172.21.15.45:60116) 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.035+0000 7fd7d9cf8640 1 -- 172.21.15.45:0/2832724447 learned_addr learned my addr 172.21.15.45:0/2832724447 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.036+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890072133 0 0) 0x7fd7d40aecb0 con 0x7fd7d414a700 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.036+0000 7fd7d94f7640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.036+0000 7fd7d94f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.036+0000 7fd7d94f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.036+0000 7fd7d94f7640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:54:39.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.036+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c0003520 con 0x7fd7d414a700 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.039+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691129551 0 0) 0x7fd7c0003520 con 0x7fd7d414a700 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.039+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 >> v1:172.21.15.45:6790/0 conn(0x7fd7d40ac5a0 legacy=0x7fd7d4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 >> v1:172.21.15.45:6789/0 conn(0x7fd7d40b0770 legacy=0x7fd7d4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: _finish_hunting 0 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 1 monclient: found mon.noname-c 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d40ac9d0 con 0x7fd7d414a700 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: _finish_auth 0 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: _check_auth_tickets 2024-03-29T10:54:39.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd7d00016b0 con 0x7fd7d414a700 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7d94f7640 10 monclient: dump: 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:39.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7dc75d640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:54:39.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.041+0000 7fd7dc75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:39.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.043+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7d0001880 con 0x7fd7d414a700 2024-03-29T10:54:39.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.043+0000 7fd7d94f7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:39.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.043+0000 7fd7d94f7640 1 -- 172.21.15.45:0/2832724447 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd7d0001c90 con 0x7fd7d414a700 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.043+0000 7fd7d94f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.043+0000 7fd7d94f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.043+0000 7fd7d94f7640 10 monclient: dump: 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:39.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.045+0000 7fd7dc75d640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:39.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.045+0000 7fd7dc75d640 4 set_mon_vals no callback set 2024-03-29T10:54:39.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.046+0000 7fd7dc75d640 10 monclient: shutdown 2024-03-29T10:54:39.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.046+0000 7fd7dc75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:39.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.046+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2832724447 >> v1:172.21.15.116:6789/0 conn(0x7fd7d414a700 legacy=0x7fd7d40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.048+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2832724447 shutdown_connections 2024-03-29T10:54:39.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.048+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2832724447 >> 172.21.15.45:0/2832724447 conn(0x7fd7d406b660 msgr2=0x7fd7d406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:39.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.048+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2832724447 shutdown_connections 2024-03-29T10:54:39.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.049+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2832724447 wait complete. 2024-03-29T10:54:39.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.049+0000 7fd7dc75d640 10 monclient: build_initial_monmap 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.049+0000 7fd7dc75d640 10 monclient: monmap: 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 1 librados: starting msgr at 2024-03-29T10:54:39.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 1 librados: starting objecter 2024-03-29T10:54:39.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 1 Processor -- start 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 1 -- start start 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 1 librados: setting wanted keys 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 1 librados: calling monclient init 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.050+0000 7fd7dc75d640 10 monclient: init 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.051+0000 7fd7dc75d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.051+0000 7fd7dc75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.051+0000 7fd7dc75d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7d40ac5a0 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7da4f9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd7d40ac5a0 0x7fd7d41450e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:60306/0 (socket says 172.21.15.45:60306) 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7da4f9640 1 -- 172.21.15.45:0/2048638590 learned_addr learned my addr 172.21.15.45:0/2048638590 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7d40b0770 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7d414a700 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d41239d0 con 0x7fd7d40b0770 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d41284f0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d4126100 con 0x7fd7d414a700 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7dc75d640 10 monclient(hunting): authenticate will time out at 953.322538s 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 client.?.objecter ms_handle_connect 0x7fd7d40ac5a0 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 client.?.objecter ms_handle_connect 0x7fd7d40b0770 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14897632 0 0) 0x7fd7d41284f0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:54:39.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.053+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b8004eb0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2216278694 0 0) 0x7fd7b8004eb0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7d41284f0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd7c4001990 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:39.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 10 monclient(hunting): dump: 2024-03-29T10:54:39.042 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:39.042 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:39.042 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.042 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3769003594 0 0) 0x7fd7d41284f0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.054+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 >> v1:172.21.15.45:6790/0 conn(0x7fd7d414a700 legacy=0x7fd7d41e7090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 >> v1:172.21.15.45:6789/0 conn(0x7fd7d40b0770 legacy=0x7fd7d4126840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 1 monclient: found mon.b 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d4127560 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 10 monclient: _renew_subs 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d41df980 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 10 monclient: _renew_subs 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d41df140 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7dc75d640 1 librados: init done 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7c4001c70 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd7c4003a60 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7caffd640 10 monclient: dump: 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.057+0000 7fd7d8cf6640 4 set_mon_vals no callback set 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.058+0000 7fd7c8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.058+0000 7fd7c8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.058+0000 7fd7c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.058+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d41dfc10 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.059+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fd7c4003bc0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fd7c4084ad0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7caffd640 10 client.4330.objecter ms_dispatch 0x7fd7d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7caffd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7caffd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7caffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7d9cf8640 10 monclient: get_auth_request con 0x7fd7b806dbd0 auth_method 0 2024-03-29T10:54:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.060+0000 7fd7caffd640 10 client.4330.objecter ms_handle_connect 0x7fd7b806dbd0 2024-03-29T10:54:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.067+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7c4086e60 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.067+0000 7fd7caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.067+0000 7fd7caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.389+0000 7fd7c8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.389+0000 7fd7c8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.389+0000 7fd7c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.389+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7d41276f0 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.390+0000 7fd7caffd640 1 -- 172.21.15.45:0/2048638590 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fd7d41dfc10 con 0x7fd7d40ac5a0 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.390+0000 7fd7caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.390+0000 7fd7caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:39.372 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:39.372 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:39.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.394+0000 7fd7c8ff9640 10 librados: watch_flush enter 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 10 librados: watch_flush exit 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 10 client.4330.objecter successfully canceled tick 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 >> v1:172.21.15.116:6816/3290579615 conn(0x7fd7b806dbd0 legacy=0x7fd7b8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 10 monclient: shutdown 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.395+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 >> v1:172.21.15.116:6789/0 conn(0x7fd7d40ac5a0 legacy=0x7fd7d41450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.396+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 shutdown_connections 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.396+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 >> 172.21.15.45:0/2048638590 conn(0x7fd7d406b660 msgr2=0x7fd7d40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.396+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 shutdown_connections 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.396+0000 7fd7c8ff9640 1 -- 172.21.15.45:0/2048638590 wait complete. 2024-03-29T10:54:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:39.396+0000 7fd7c8ff9640 1 librados: shutdown 2024-03-29T10:54:39.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:46.202 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.367+0000 7fdcda406640 10 monclient: get_monmap_and_config 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.367+0000 7fdcda406640 10 monclient: build_initial_monmap 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.367+0000 7fdcda406640 10 monclient: monmap: 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:46.368097+0000 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:46.368097+0000 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.348 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:46.349 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:46.349 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:46.349 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.369+0000 7fdcda406640 1 Processor -- start 2024-03-29T10:54:46.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.369+0000 7fdcda406640 1 -- start start 2024-03-29T10:54:46.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.369+0000 7fdcda406640 10 monclient: init 2024-03-29T10:54:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient(hunting): picked mon.noname-c con 0x7fdcd41422b0 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient(hunting): picked mon.noname-b con 0x7fdcd4142690 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient(hunting): picked mon.noname-a con 0x7fdcd40b17d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd4142c80 con 0x7fdcd40b17d0 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd40aae40 con 0x7fdcd41422b0 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd413fbd0 con 0x7fdcd4142690 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.370+0000 7fdcda406640 10 monclient(hunting): authenticate will time out at 960.639109s 2024-03-29T10:54:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.372+0000 7fdcd89a3640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdcd40b17d0 0x7fdcd40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39665/0 (socket says 172.21.15.45:39665) 2024-03-29T10:54:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.372+0000 7fdcd3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdcd41422b0 0x7fdcd413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:47476/0 (socket says 172.21.15.45:47476) 2024-03-29T10:54:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.373+0000 7fdcd3fff640 1 -- 172.21.15.45:0/483171624 learned_addr learned my addr 172.21.15.45:0/483171624 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435175344 0 0) 0x7fdcd40aae40 con 0x7fdcd41422b0 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcbc003520 con 0x7fdcd41422b0 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314129328 0 0) 0x7fdcd4142c80 con 0x7fdcd40b17d0 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:54:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd40aae40 con 0x7fdcd40b17d0 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613826115 0 0) 0x7fdcd40aae40 con 0x7fdcd40b17d0 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 >> v1:172.21.15.45:6790/0 conn(0x7fdcd4142690 legacy=0x7fdcd40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 >> v1:172.21.15.116:6789/0 conn(0x7fdcd41422b0 legacy=0x7fdcd413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 monclient: found mon.noname-a 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4151940 con 0x7fdcd40b17d0 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.374+0000 7fdcd2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:46.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdcc8001670 con 0x7fdcd40b17d0 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: dump: 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcc8001890 con 0x7fdcd40b17d0 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 1 -- 172.21.15.45:0/483171624 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdcc8001ba0 con 0x7fdcd40b17d0 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcd2ffd640 10 monclient: dump: 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:46.358 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:46.359 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcda406640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:54:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcda406640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.375+0000 7fdcda406640 4 set_mon_vals no callback set 2024-03-29T10:54:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.376+0000 7fdcda406640 10 monclient: shutdown 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.376+0000 7fdcda406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.376+0000 7fdcda406640 1 -- 172.21.15.45:0/483171624 >> v1:172.21.15.45:6789/0 conn(0x7fdcd40b17d0 legacy=0x7fdcd40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.376+0000 7fdcda406640 1 -- 172.21.15.45:0/483171624 shutdown_connections 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.376+0000 7fdcda406640 1 -- 172.21.15.45:0/483171624 >> 172.21.15.45:0/483171624 conn(0x7fdcd4066a00 msgr2=0x7fdcd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.377+0000 7fdcda406640 1 -- 172.21.15.45:0/483171624 shutdown_connections 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.377+0000 7fdcda406640 1 -- 172.21.15.45:0/483171624 wait complete. 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.378+0000 7fdcda406640 10 monclient: build_initial_monmap 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.378+0000 7fdcda406640 10 monclient: monmap: 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.361 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:46.362 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:46.362 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:46.362 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.378+0000 7fdcda406640 1 librados: starting msgr at 2024-03-29T10:54:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.378+0000 7fdcda406640 1 librados: starting objecter 2024-03-29T10:54:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.378+0000 7fdcda406640 1 Processor -- start 2024-03-29T10:54:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.381+0000 7fdcda406640 1 -- start start 2024-03-29T10:54:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.381+0000 7fdcda406640 1 librados: setting wanted keys 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.381+0000 7fdcda406640 1 librados: calling monclient init 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.381+0000 7fdcda406640 10 monclient: init 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.381+0000 7fdcda406640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 10 monclient(hunting): picked mon.noname-a con 0x7fdcd40b17d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 10 monclient(hunting): picked mon.noname-b con 0x7fdcd41422b0 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 10 monclient(hunting): picked mon.noname-c con 0x7fdcd4142690 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd406de90 con 0x7fdcd40b17d0 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd406e0e0 con 0x7fdcd41422b0 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd406e850 con 0x7fdcd4142690 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.382+0000 7fdcda406640 10 monclient(hunting): authenticate will time out at 960.651774s 2024-03-29T10:54:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.383+0000 7fdcd37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdcd41422b0 0x7fdcd412ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:47610/0 (socket says 172.21.15.45:47610) 2024-03-29T10:54:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.383+0000 7fdcd37fe640 1 -- 172.21.15.45:0/154542015 learned_addr learned my addr 172.21.15.45:0/154542015 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.385+0000 7fdcd0ff9640 10 client.?.objecter ms_handle_connect 0x7fdcd40b17d0 2024-03-29T10:54:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.385+0000 7fdcd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.385+0000 7fdcd0ff9640 10 client.?.objecter ms_handle_connect 0x7fdcd41422b0 2024-03-29T10:54:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.385+0000 7fdcd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.386+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052532604 0 0) 0x7fdcd406e0e0 con 0x7fdcd41422b0 2024-03-29T10:54:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.386+0000 7fdcd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.386+0000 7fdcd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.386+0000 7fdcd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.386+0000 7fdcd0ff9640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:54:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.386+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0004eb0 con 0x7fdcd41422b0 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346616639 0 0) 0x7fdcd406de90 con 0x7fdcd40b17d0 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd406e0e0 con 0x7fdcd40b17d0 2024-03-29T10:54:46.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481620040 0 0) 0x7fdcd406e0e0 con 0x7fdcd40b17d0 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcd406de90 con 0x7fdcd40b17d0 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 client.?.objecter ms_handle_connect 0x7fdcd4142690 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdcc4001880 con 0x7fdcd40b17d0 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:46.373 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764525916 0 0) 0x7fdcc0004eb0 con 0x7fdcd41422b0 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcd406e0e0 con 0x7fdcd41422b0 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.387+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346483668 0 0) 0x7fdcd406e850 con 0x7fdcd4142690 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0004eb0 con 0x7fdcd4142690 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdccc001ad0 con 0x7fdcd41422b0 2024-03-29T10:54:46.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 430622721 0 0) 0x7fdcd406de90 con 0x7fdcd40b17d0 2024-03-29T10:54:46.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 >> v1:172.21.15.45:6790/0 conn(0x7fdcd4142690 legacy=0x7fdcd41e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 >> v1:172.21.15.116:6789/0 conn(0x7fdcd41422b0 legacy=0x7fdcd412ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 monclient: found mon.a 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd406eb30 con 0x7fdcd40b17d0 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 5 monclient: authenticate success, global_id 4329 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 10 monclient: _renew_subs 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd406ee10 con 0x7fdcd40b17d0 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:46.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 10 monclient: _renew_subs 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd412f430 con 0x7fdcd40b17d0 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.388+0000 7fdcda406640 1 librados: init done 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.389+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcc4001d70 con 0x7fdcd40b17d0 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.389+0000 7fdcd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.389+0000 7fdcd27fc640 4 set_mon_vals no callback set 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdcc4004980 con 0x7fdcd40b17d0 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcd0ff9640 10 monclient: dump: 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:46.377 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcda406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcda406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcda406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:46.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.390+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb4005080 con 0x7fdcd40b17d0 2024-03-29T10:54:46.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fdcc4004ae0 con 0x7fdcd40b17d0 2024-03-29T10:54:46.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fdcc4084970 con 0x7fdcd40b17d0 2024-03-29T10:54:46.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd0ff9640 10 client.4329.objecter ms_dispatch 0x7fdcd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:46.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd0ff9640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:46.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd0ff9640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:46.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd0ff9640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T10:54:46.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.394+0000 7fdcd37fe640 10 monclient: get_auth_request con 0x7fdcc006df80 auth_method 0 2024-03-29T10:54:46.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.395+0000 7fdcd0ff9640 10 client.4329.objecter ms_handle_connect 0x7fdcc006df80 2024-03-29T10:54:46.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.401+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcc4050250 con 0x7fdcd40b17d0 2024-03-29T10:54:46.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.401+0000 7fdcd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:46.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.401+0000 7fdcd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:46.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.732+0000 7fdcda406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:46.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.732+0000 7fdcda406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:46.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.732+0000 7fdcda406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:54:46.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.732+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcb4005330 con 0x7fdcd40b17d0 2024-03-29T10:54:46.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.733+0000 7fdcd0ff9640 1 -- 172.21.15.45:0/154542015 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fdcc4052180 con 0x7fdcd40b17d0 2024-03-29T10:54:46.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.733+0000 7fdcd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:46.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.733+0000 7fdcd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:46.714 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:46.714 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:46.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.736+0000 7fdcda406640 10 librados: watch_flush enter 2024-03-29T10:54:46.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 10 librados: watch_flush exit 2024-03-29T10:54:46.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T10:54:46.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 10 client.4329.objecter successfully canceled tick 2024-03-29T10:54:46.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 >> v1:172.21.15.116:6816/3290579615 conn(0x7fdcc006df80 legacy=0x7fdcc0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 10 monclient: shutdown 2024-03-29T10:54:46.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:46.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 >> v1:172.21.15.45:6789/0 conn(0x7fdcd40b17d0 legacy=0x7fdcd40b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 shutdown_connections 2024-03-29T10:54:46.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 >> 172.21.15.45:0/154542015 conn(0x7fdcd4066a00 msgr2=0x7fdcd413c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:46.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 shutdown_connections 2024-03-29T10:54:46.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 -- 172.21.15.45:0/154542015 wait complete. 2024-03-29T10:54:46.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:46.737+0000 7fdcda406640 1 librados: shutdown 2024-03-29T10:54:46.735 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:53.541 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:53.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.698+0000 7f6df63de640 10 monclient: get_monmap_and_config 2024-03-29T10:54:53.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.698+0000 7f6df63de640 10 monclient: build_initial_monmap 2024-03-29T10:54:53.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.698+0000 7f6df63de640 10 monclient: monmap: 2024-03-29T10:54:53.679 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:54:53.699311+0000 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:54:53.699311+0000 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:54:53.680 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:53.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.700+0000 7f6df63de640 1 Processor -- start 2024-03-29T10:54:53.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.701+0000 7f6df63de640 1 -- start start 2024-03-29T10:54:53.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.701+0000 7f6df63de640 10 monclient: init 2024-03-29T10:54:53.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:53.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:53.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient(hunting): picked mon.noname-a con 0x7f6de80389e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:53.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient(hunting): picked mon.noname-b con 0x7f6de8038dc0 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:53.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient(hunting): picked mon.noname-c con 0x7f6de8030660 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:53.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de80c54c0 con 0x7f6de80389e0 2024-03-29T10:54:53.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de8036430 con 0x7f6de8030660 2024-03-29T10:54:53.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de8039840 con 0x7f6de8038dc0 2024-03-29T10:54:53.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:53.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.702+0000 7f6df63de640 10 monclient(hunting): authenticate will time out at 967.971177s 2024-03-29T10:54:53.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.703+0000 7f6df5bdd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6de8030660 0x7f6de80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:56326/0 (socket says 172.21.15.45:56326) 2024-03-29T10:54:53.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.703+0000 7f6df5bdd640 1 -- 172.21.15.45:0/564744295 learned_addr learned my addr 172.21.15.45:0/564744295 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:53.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682442751 0 0) 0x7f6de8036430 con 0x7f6de8030660 2024-03-29T10:54:53.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:53.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:53.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:53.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:54:53.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc003520 con 0x7f6de8030660 2024-03-29T10:54:53.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385555390 0 0) 0x7f6ddc003520 con 0x7f6de8030660 2024-03-29T10:54:53.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.705+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 >> v1:172.21.15.45:6790/0 conn(0x7f6de8038dc0 legacy=0x7f6de80cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:53.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.706+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 >> v1:172.21.15.45:6789/0 conn(0x7f6de80389e0 legacy=0x7f6de8033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:53.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:53.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 1 monclient: found mon.noname-c 2024-03-29T10:54:53.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de8035690 con 0x7f6de8030660 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6de7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6df63de640 5 monclient: authenticate success, global_id 4339 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.708+0000 7f6df63de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6dec001690 con 0x7f6de8030660 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6de7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:53.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6de7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6de7fff640 10 monclient: dump: 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6df63de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:53.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dec001920 con 0x7f6de8030660 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6de7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6df63de640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.709+0000 7f6df63de640 4 set_mon_vals no callback set 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6de7fff640 1 -- 172.21.15.45:0/564744295 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6dec001cb0 con 0x7f6de8030660 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6de7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6de7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6de7fff640 10 monclient: dump: 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:53.694 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 10 monclient: shutdown 2024-03-29T10:54:53.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:53.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 1 -- 172.21.15.45:0/564744295 >> v1:172.21.15.116:6789/0 conn(0x7f6de8030660 legacy=0x7f6de80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:53.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 1 -- 172.21.15.45:0/564744295 shutdown_connections 2024-03-29T10:54:53.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 1 -- 172.21.15.45:0/564744295 >> 172.21.15.45:0/564744295 conn(0x7f6de8005c00 msgr2=0x7f6de8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:53.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 1 -- 172.21.15.45:0/564744295 shutdown_connections 2024-03-29T10:54:53.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.710+0000 7f6df63de640 1 -- 172.21.15.45:0/564744295 wait complete. 2024-03-29T10:54:53.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.711+0000 7f6df63de640 10 monclient: build_initial_monmap 2024-03-29T10:54:53.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.711+0000 7f6df63de640 10 monclient: monmap: 2024-03-29T10:54:53.712 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:54:53.712 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:53.712 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:54:53.712 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:54:53.713 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:53.713 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:53.713 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:54:53.713 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:54:53.713 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:54:53.713 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.711+0000 7f6df63de640 1 librados: starting msgr at 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.711+0000 7f6df63de640 1 librados: starting objecter 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.711+0000 7f6df63de640 1 Processor -- start 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 1 -- start start 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 1 librados: setting wanted keys 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 1 librados: calling monclient init 2024-03-29T10:54:53.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 10 monclient: init 2024-03-29T10:54:53.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:53.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:53.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 10 monclient(hunting): picked mon.noname-c con 0x7f6de8030660 addr v1:172.21.15.45:6790/0 2024-03-29T10:54:53.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 10 monclient(hunting): picked mon.noname-a con 0x7f6de80389e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:54:53.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 10 monclient(hunting): picked mon.noname-b con 0x7f6de80c7d90 addr v1:172.21.15.116:6789/0 2024-03-29T10:54:53.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de80c8420 con 0x7f6de80389e0 2024-03-29T10:54:53.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.712+0000 7f6df63de640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de80cab00 con 0x7f6de80c7d90 2024-03-29T10:54:53.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.713+0000 7f6df5bdd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6de80c7d90 0x7f6de80ca3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:56392/0 (socket says 172.21.15.45:56392) 2024-03-29T10:54:53.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.713+0000 7f6df5bdd640 1 -- 172.21.15.45:0/2583971047 learned_addr learned my addr 172.21.15.45:0/2583971047 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.713+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de802f680 con 0x7f6de8030660 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.714+0000 7f6df63de640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.714+0000 7f6df63de640 10 monclient(hunting): authenticate will time out at 967.983175s 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.716+0000 7f6de5ffb640 10 client.?.objecter ms_handle_connect 0x7f6de8030660 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.716+0000 7f6de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.716+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977277743 0 0) 0x7f6de802f680 con 0x7f6de8030660 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:53.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd4004eb0 con 0x7f6de8030660 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 10 client.?.objecter ms_handle_connect 0x7f6de80c7d90 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.717+0000 7f6de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.718+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851489013 0 0) 0x7f6de80cab00 con 0x7f6de80c7d90 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.718+0000 7f6de5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:53.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.718+0000 7f6de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.718+0000 7f6de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.718+0000 7f6de5ffb640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.718+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de802f680 con 0x7f6de80c7d90 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3651787336 0 0) 0x7f6dd4004eb0 con 0x7f6de8030660 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de80cab00 con 0x7f6de8030660 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6de0001820 con 0x7f6de8030660 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:53.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:53.720 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:53.721 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:53.721 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:53.721 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:53.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:53.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4230925666 0 0) 0x7f6de80cab00 con 0x7f6de8030660 2024-03-29T10:54:53.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 >> v1:172.21.15.116:6789/0 conn(0x7f6de80c7d90 legacy=0x7f6de80ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.719+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 >> v1:172.21.15.45:6789/0 conn(0x7f6de80389e0 legacy=0x7f6de80c9ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 1 monclient: found mon.c 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de802f810 con 0x7f6de8030660 2024-03-29T10:54:53.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:53.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:53.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6de5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:53.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:54:53.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 10 monclient: _renew_subs 2024-03-29T10:54:53.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:53.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de802fa00 con 0x7f6de8030660 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 10 monclient: _renew_subs 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de80c7920 con 0x7f6de8030660 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.720+0000 7f6df63de640 1 librados: init done 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6de0001980 con 0x7f6de8030660 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6de00036e0 con 0x7f6de8030660 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:53.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de77fe640 4 set_mon_vals no callback set 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6de5ffb640 10 monclient: dump: 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:54:53.725 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.721+0000 7f6df63de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.722+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f6de0019770 con 0x7f6de8030660 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.722+0000 7f6df63de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.722+0000 7f6df63de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.722+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd0005080 con 0x7f6de8030660 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.722+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f6de00847c0 con 0x7f6de8030660 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.723+0000 7f6de5ffb640 10 client.4325.objecter ms_dispatch 0x7f6df0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.723+0000 7f6de5ffb640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.723+0000 7f6de5ffb640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.723+0000 7f6de5ffb640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T10:54:53.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.724+0000 7f6df4bdb640 10 monclient: get_auth_request con 0x7f6dd406dd30 auth_method 0 2024-03-29T10:54:53.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.724+0000 7f6de5ffb640 10 client.4325.objecter ms_handle_connect 0x7f6dd406dd30 2024-03-29T10:54:53.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.733+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6de004f070 con 0x7f6de8030660 2024-03-29T10:54:53.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.733+0000 7f6de5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:53.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:53.733+0000 7f6de5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:54.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.039+0000 7f6df63de640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.039+0000 7f6df63de640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.039+0000 7f6df63de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:54:54.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.039+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dd0005330 con 0x7f6de8030660 2024-03-29T10:54:54.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.040+0000 7f6de5ffb640 1 -- 172.21.15.45:0/2583971047 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f6de0050fa0 con 0x7f6de8030660 2024-03-29T10:54:54.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.040+0000 7f6de5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.040+0000 7f6de5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:54.022 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:54:54.022 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.043+0000 7f6df63de640 10 librados: watch_flush enter 2024-03-29T10:54:54.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.044+0000 7f6df63de640 10 librados: watch_flush exit 2024-03-29T10:54:54.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.044+0000 7f6df63de640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T10:54:54.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.044+0000 7f6df63de640 10 client.4325.objecter successfully canceled tick 2024-03-29T10:54:54.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 >> v1:172.21.15.116:6816/3290579615 conn(0x7f6dd406dd30 legacy=0x7f6dd40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 10 monclient: shutdown 2024-03-29T10:54:54.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:54.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 >> v1:172.21.15.45:6790/0 conn(0x7f6de8030660 legacy=0x7f6de8160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 shutdown_connections 2024-03-29T10:54:54.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 >> 172.21.15.45:0/2583971047 conn(0x7f6de8005c00 msgr2=0x7f6de8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:54.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 shutdown_connections 2024-03-29T10:54:54.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.047+0000 7f6df63de640 1 -- 172.21.15.45:0/2583971047 wait complete. 2024-03-29T10:54:54.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:54:54.048+0000 7f6df63de640 1 librados: shutdown 2024-03-29T10:54:54.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:00.850 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:00.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.010+0000 7f69393c1640 10 monclient: get_monmap_and_config 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.010+0000 7f69393c1640 10 monclient: build_initial_monmap 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.011+0000 7f69393c1640 10 monclient: monmap: 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:01.011865+0000 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:01.011865+0000 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:00.992 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:00.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.012+0000 7f69393c1640 1 Processor -- start 2024-03-29T10:55:00.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.012+0000 7f69393c1640 1 -- start start 2024-03-29T10:55:00.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.012+0000 7f69393c1640 10 monclient: init 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.012+0000 7f69393c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.012+0000 7f69393c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.012+0000 7f69393c1640 10 monclient(hunting): picked mon.noname-b con 0x7f69341457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 10 monclient(hunting): picked mon.noname-a con 0x7f693413f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 10 monclient(hunting): picked mon.noname-c con 0x7f69340a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69340aaef0 con 0x7f693413f650 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934145bd0 con 0x7f69340a9d70 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934145eb0 con 0x7f69341457f0 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69337fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f69340a9d70 0x7f693414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:42602/0 (socket says 172.21.15.45:42602) 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69337fe640 1 -- 172.21.15.45:0/341169438 learned_addr learned my addr 172.21.15.45:0/341169438 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:00.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.013+0000 7f69393c1640 10 monclient(hunting): authenticate will time out at 975.282842s 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.017+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934719096 0 0) 0x7f6934145bd0 con 0x7f69340a9d70 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.017+0000 7f69327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.017+0000 7f69327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.017+0000 7f69327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.017+0000 7f69327fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.018+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920003520 con 0x7f69340a9d70 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.019+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2791268192 0 0) 0x7f6920003520 con 0x7f69340a9d70 2024-03-29T10:55:00.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.019+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 >> v1:172.21.15.45:6790/0 conn(0x7f69341457f0 legacy=0x7f693413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:01.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 >> v1:172.21.15.45:6789/0 conn(0x7f693413f650 legacy=0x7f69340b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:01.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:01.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 1 monclient: found mon.noname-c 2024-03-29T10:55:01.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:01.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:55:01.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934146340 con 0x7f69340a9d70 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f692c001a10 con 0x7f69340a9d70 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: dump: 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:01.003 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69393c1640 5 monclient: authenticate success, global_id 4345 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69393c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f692c001be0 con 0x7f69340a9d70 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69393c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 1 -- 172.21.15.45:0/341169438 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f692c002030 con 0x7f69340a9d70 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69393c1640 4 set_mon_vals no callback set 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.020+0000 7f69327fc640 10 monclient: dump: 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.004 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.021+0000 7f69393c1640 10 monclient: shutdown 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.021+0000 7f69393c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.021+0000 7f69393c1640 1 -- 172.21.15.45:0/341169438 >> v1:172.21.15.116:6789/0 conn(0x7f69340a9d70 legacy=0x7f693414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.021+0000 7f69393c1640 1 -- 172.21.15.45:0/341169438 shutdown_connections 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.021+0000 7f69393c1640 1 -- 172.21.15.45:0/341169438 >> 172.21.15.45:0/341169438 conn(0x7f6934066a00 msgr2=0x7f6934066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:01.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.022+0000 7f69393c1640 1 -- 172.21.15.45:0/341169438 shutdown_connections 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.022+0000 7f69393c1640 1 -- 172.21.15.45:0/341169438 wait complete. 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.023+0000 7f69393c1640 10 monclient: build_initial_monmap 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.023+0000 7f69393c1640 10 monclient: monmap: 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:01.007 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 1 librados: starting msgr at 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 1 librados: starting objecter 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 1 Processor -- start 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 1 -- start start 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 1 librados: setting wanted keys 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 1 librados: calling monclient init 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.024+0000 7f69393c1640 10 monclient: init 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.025+0000 7f69393c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.025+0000 7f69393c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.025+0000 7f69393c1640 10 monclient(hunting): picked mon.noname-b con 0x7f69340a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 10 monclient(hunting): picked mon.noname-c con 0x7f693413f650 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 10 monclient(hunting): picked mon.noname-a con 0x7f69341457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934142a80 con 0x7f69341457f0 2024-03-29T10:55:01.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934131d50 con 0x7f69340a9d70 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934071290 con 0x7f693413f650 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f69393c1640 10 monclient(hunting): authenticate will time out at 975.295125s 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f6932ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f69340a9d70 0x7f69340716b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:42734/0 (socket says 172.21.15.45:42734) 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.026+0000 7f6932ffd640 1 -- 172.21.15.45:0/692865226 learned_addr learned my addr 172.21.15.45:0/692865226 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.027+0000 7f692b7fe640 10 client.?.objecter ms_handle_connect 0x7f69340a9d70 2024-03-29T10:55:01.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.027+0000 7f692b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.031+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95530535 0 0) 0x7f6934131d50 con 0x7f69340a9d70 2024-03-29T10:55:01.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.031+0000 7f692b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.031+0000 7f692b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.031+0000 7f692b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.031+0000 7f692b7fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:55:01.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.031+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6914004eb0 con 0x7f69340a9d70 2024-03-29T10:55:01.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.033+0000 7f692b7fe640 10 client.?.objecter ms_handle_connect 0x7f693413f650 2024-03-29T10:55:01.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.033+0000 7f692b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634442668 0 0) 0x7f6934071290 con 0x7f693413f650 2024-03-29T10:55:01.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:55:01.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6934131d50 con 0x7f693413f650 2024-03-29T10:55:01.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3560390323 0 0) 0x7f6934131d50 con 0x7f693413f650 2024-03-29T10:55:01.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6934071290 con 0x7f693413f650 2024-03-29T10:55:01.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f691c001860 con 0x7f693413f650 2024-03-29T10:55:01.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.034+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3722648124 0 0) 0x7f6914004eb0 con 0x7f69340a9d70 2024-03-29T10:55:01.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6934131d50 con 0x7f69340a9d70 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f69240017d0 con 0x7f69340a9d70 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3474289693 0 0) 0x7f6934071290 con 0x7f693413f650 2024-03-29T10:55:01.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 >> v1:172.21.15.116:6789/0 conn(0x7f69340a9d70 legacy=0x7f69340716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 >> v1:172.21.15.45:6789/0 conn(0x7f69341457f0 legacy=0x7f69341e6540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 monclient: found mon.c 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934142560 con 0x7f693413f650 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 10 monclient: _renew_subs 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f691c001d70 con 0x7f693413f650 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69341306e0 con 0x7f693413f650 2024-03-29T10:55:01.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 10 monclient: _renew_subs 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6934071fa0 con 0x7f693413f650 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.035+0000 7f69393c1640 1 librados: init done 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f692b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f691c003880 con 0x7f693413f650 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f692b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f692b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f6931ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f692b7fe640 10 monclient: dump: 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:01.021 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.022 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:01.022 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:01.022 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:01.022 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:01.022 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:01.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.036+0000 7f69393c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.037+0000 7f69393c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.037+0000 7f69393c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.037+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6918005080 con 0x7f693413f650 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.038+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f691c019850 con 0x7f693413f650 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.038+0000 7f6932ffd640 10 monclient: get_auth_request con 0x7f691406de80 auth_method 0 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.038+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f691c0836e0 con 0x7f693413f650 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.039+0000 7f692b7fe640 10 client.4328.objecter ms_dispatch 0x7f6934000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.039+0000 7f692b7fe640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.039+0000 7f692b7fe640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.039+0000 7f692b7fe640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T10:55:01.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.039+0000 7f692b7fe640 10 client.4328.objecter ms_handle_connect 0x7f691406de80 2024-03-29T10:55:01.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.046+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f691c04efc0 con 0x7f693413f650 2024-03-29T10:55:01.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.046+0000 7f692b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.046+0000 7f692b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.347+0000 7f69393c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.347+0000 7f69393c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.347+0000 7f69393c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:01.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.347+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6918005330 con 0x7f693413f650 2024-03-29T10:55:01.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.349+0000 7f692b7fe640 1 -- 172.21.15.45:0/692865226 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f691c050ef0 con 0x7f693413f650 2024-03-29T10:55:01.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.349+0000 7f692b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:01.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.349+0000 7f692b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:01.330 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:01.330 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.353+0000 7f69393c1640 10 librados: watch_flush enter 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.353+0000 7f69393c1640 10 librados: watch_flush exit 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.353+0000 7f69393c1640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.353+0000 7f69393c1640 10 client.4328.objecter successfully canceled tick 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 >> v1:172.21.15.116:6816/3290579615 conn(0x7f691406de80 legacy=0x7f6914070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 10 monclient: shutdown 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 >> v1:172.21.15.45:6790/0 conn(0x7f693413f650 legacy=0x7f69341e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 shutdown_connections 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 >> 172.21.15.45:0/692865226 conn(0x7f6934066a00 msgr2=0x7f69340b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 shutdown_connections 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 -- 172.21.15.45:0/692865226 wait complete. 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:01.354+0000 7f69393c1640 1 librados: shutdown 2024-03-29T10:55:01.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:08.157 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.322+0000 7f563887b640 10 monclient: get_monmap_and_config 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.322+0000 7f563887b640 10 monclient: build_initial_monmap 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.322+0000 7f563887b640 10 monclient: monmap: 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:08.323345+0000 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:08.323345+0000 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:08.303 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.323+0000 7f563887b640 1 Processor -- start 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.323+0000 7f563887b640 1 -- start start 2024-03-29T10:55:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.323+0000 7f563887b640 10 monclient: init 2024-03-29T10:55:08.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:08.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:08.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient(hunting): picked mon.noname-a con 0x7f56341422b0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient(hunting): picked mon.noname-c con 0x7f5634142690 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient(hunting): picked mon.noname-b con 0x7f56340b17d0 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5634142c80 con 0x7f56341422b0 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340aae40 con 0x7f5634142690 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563413fbd0 con 0x7f56340b17d0 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:08.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.324+0000 7f563887b640 10 monclient(hunting): authenticate will time out at 982.593853s 2024-03-29T10:55:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.330+0000 7f5632d9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f56340b17d0 0x7f56340b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46679/0 (socket says 172.21.15.45:46679) 2024-03-29T10:55:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.330+0000 7f5632d9d640 1 -- 172.21.15.45:0/3743293502 learned_addr learned my addr 172.21.15.45:0/3743293502 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.330+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143310948 0 0) 0x7f563413fbd0 con 0x7f56340b17d0 2024-03-29T10:55:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561c003520 con 0x7f56340b17d0 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924113580 0 0) 0x7f5634142c80 con 0x7f56341422b0 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f563413fbd0 con 0x7f56341422b0 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412797100 0 0) 0x7f563413fbd0 con 0x7f56341422b0 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 >> v1:172.21.15.45:6790/0 conn(0x7f56340b17d0 legacy=0x7f56340b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.331+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 >> v1:172.21.15.116:6789/0 conn(0x7f5634142690 legacy=0x7f56340b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 1 monclient: found mon.noname-a 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634151940 con 0x7f56341422b0 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: _finish_auth 0 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563887b640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563887b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f562c0019f0 con 0x7f56341422b0 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: dump: 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563887b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:08.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f562c001b50 con 0x7f56341422b0 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563887b640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563887b640 4 set_mon_vals no callback set 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 1 -- 172.21.15.45:0/3743293502 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f562c001f20 con 0x7f56341422b0 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.332+0000 7f563159a640 10 monclient: dump: 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:08.317 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.333+0000 7f563887b640 10 monclient: shutdown 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.333+0000 7f563887b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.333+0000 7f563887b640 1 -- 172.21.15.45:0/3743293502 >> v1:172.21.15.45:6789/0 conn(0x7f56341422b0 legacy=0x7f563413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.334+0000 7f563887b640 1 -- 172.21.15.45:0/3743293502 shutdown_connections 2024-03-29T10:55:08.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.334+0000 7f563887b640 1 -- 172.21.15.45:0/3743293502 >> 172.21.15.45:0/3743293502 conn(0x7f5634066a00 msgr2=0x7f5634066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.334+0000 7f563887b640 1 -- 172.21.15.45:0/3743293502 shutdown_connections 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.334+0000 7f563887b640 1 -- 172.21.15.45:0/3743293502 wait complete. 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.336+0000 7f563887b640 10 monclient: build_initial_monmap 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.336+0000 7f563887b640 10 monclient: monmap: 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:08.320 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.336+0000 7f563887b640 1 librados: starting msgr at 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.336+0000 7f563887b640 1 librados: starting objecter 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.336+0000 7f563887b640 1 Processor -- start 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.337+0000 7f563887b640 1 -- start start 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.337+0000 7f563887b640 1 librados: setting wanted keys 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.337+0000 7f563887b640 1 librados: calling monclient init 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.337+0000 7f563887b640 10 monclient: init 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient(hunting): picked mon.noname-a con 0x7f56340b17d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient(hunting): picked mon.noname-c con 0x7f56341422b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient(hunting): picked mon.noname-b con 0x7f5634142690 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341dfce0 con 0x7f56340b17d0 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5634139c40 con 0x7f5634142690 2024-03-29T10:55:08.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341df890 con 0x7f56341422b0 2024-03-29T10:55:08.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:08.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.338+0000 7f563887b640 10 monclient(hunting): authenticate will time out at 982.607720s 2024-03-29T10:55:08.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.339+0000 7f5632d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5634142690 0x7f563413a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:45414/0 (socket says 172.21.15.45:45414) 2024-03-29T10:55:08.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.339+0000 7f5632d9d640 1 -- 172.21.15.45:0/3397362924 learned_addr learned my addr 172.21.15.45:0/3397362924 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:08.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.340+0000 7f5622ffd640 10 client.?.objecter ms_handle_connect 0x7f5634142690 2024-03-29T10:55:08.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.340+0000 7f5622ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:08.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 10 client.?.objecter ms_handle_connect 0x7f56340b17d0 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901651470 0 0) 0x7f56341dfce0 con 0x7f56340b17d0 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.342+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5614004eb0 con 0x7f56340b17d0 2024-03-29T10:55:08.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099090918 0 0) 0x7f5634139c40 con 0x7f5634142690 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56140051d0 con 0x7f5634142690 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610453490 0 0) 0x7f5614004eb0 con 0x7f56340b17d0 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5614005640 con 0x7f56340b17d0 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f562c001e60 con 0x7f56340b17d0 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:08.325 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1017090828 0 0) 0x7f5614005640 con 0x7f56340b17d0 2024-03-29T10:55:08.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.343+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 >> v1:172.21.15.45:6790/0 conn(0x7f56341422b0 legacy=0x7f5634070690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 >> v1:172.21.15.116:6789/0 conn(0x7f5634142690 legacy=0x7f563413a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 1 monclient: found mon.a 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56341dfa20 con 0x7f56340b17d0 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f5622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f563887b640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f563887b640 10 monclient: _renew_subs 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f563887b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.344+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634070da0 con 0x7f56340b17d0 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.345+0000 7f563887b640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:08.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.345+0000 7f563887b640 10 monclient: _renew_subs 2024-03-29T10:55:08.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.345+0000 7f563887b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:08.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.345+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5634071310 con 0x7f56340b17d0 2024-03-29T10:55:08.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.345+0000 7f563887b640 1 librados: init done 2024-03-29T10:55:08.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.346+0000 7f563887b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:08.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.346+0000 7f563887b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:08.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.346+0000 7f563887b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.346+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5634075f10 con 0x7f56340b17d0 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.347+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f562c003c20 con 0x7f56340b17d0 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.347+0000 7f5622ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.347+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f562c001e60 con 0x7f56340b17d0 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.347+0000 7f5622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.347+0000 7f5622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.347+0000 7f5622ffd640 10 monclient: dump: 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:08.329 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:08.330 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:08.330 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:08.330 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:08.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.348+0000 7f5630d99640 4 set_mon_vals no callback set 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.348+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f562c019c00 con 0x7f56340b17d0 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.349+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f562c083920 con 0x7f56340b17d0 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.349+0000 7f5622ffd640 10 client.4335.objecter ms_dispatch 0x7f5634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.349+0000 7f5622ffd640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.349+0000 7f5622ffd640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.349+0000 7f5631d9b640 10 monclient: get_auth_request con 0x7f561406e370 auth_method 0 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.349+0000 7f5622ffd640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-29T10:55:08.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.350+0000 7f5622ffd640 10 client.4335.objecter ms_handle_connect 0x7f561406e370 2024-03-29T10:55:08.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.359+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f562c04f260 con 0x7f56340b17d0 2024-03-29T10:55:08.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.359+0000 7f5622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:08.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.359+0000 7f5622ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:08.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.683+0000 7f563887b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:08.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.683+0000 7f563887b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:08.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.683+0000 7f563887b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:08.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.683+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5634072040 con 0x7f56340b17d0 2024-03-29T10:55:08.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.686+0000 7f5622ffd640 1 -- 172.21.15.45:0/3397362924 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f562c082fd0 con 0x7f56340b17d0 2024-03-29T10:55:08.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.686+0000 7f5622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:08.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.686+0000 7f5622ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:08.667 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:08.667 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:08.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.689+0000 7f563887b640 10 librados: watch_flush enter 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 10 librados: watch_flush exit 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 10 client.4335.objecter successfully canceled tick 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 >> v1:172.21.15.116:6816/3290579615 conn(0x7f561406e370 legacy=0x7f5614070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 10 monclient: shutdown 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 >> v1:172.21.15.45:6789/0 conn(0x7f56340b17d0 legacy=0x7f56341df180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.692+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 shutdown_connections 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.693+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 >> 172.21.15.45:0/3397362924 conn(0x7f5634066a00 msgr2=0x7f56340b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:08.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.697+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 shutdown_connections 2024-03-29T10:55:08.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.697+0000 7f563887b640 1 -- 172.21.15.45:0/3397362924 wait complete. 2024-03-29T10:55:08.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:08.697+0000 7f563887b640 1 librados: shutdown 2024-03-29T10:55:08.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:15.502 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.680+0000 7f60ed93b640 10 monclient: get_monmap_and_config 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.680+0000 7f60ed93b640 10 monclient: build_initial_monmap 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.680+0000 7f60ed93b640 10 monclient: monmap: 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:15.681250+0000 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:15.681250+0000 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:15.661 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:15.662 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:15.662 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:15.662 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:15.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.681+0000 7f60ed93b640 1 Processor -- start 2024-03-29T10:55:15.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.681+0000 7f60ed93b640 1 -- start start 2024-03-29T10:55:15.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.681+0000 7f60ed93b640 10 monclient: init 2024-03-29T10:55:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.682+0000 7f60ed93b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.682+0000 7f60ed93b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 10 monclient(hunting): picked mon.noname-b con 0x7f60e80b2800 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 10 monclient(hunting): picked mon.noname-a con 0x7f60e80b3220 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 10 monclient(hunting): picked mon.noname-c con 0x7f60e81439c0 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e80b3840 con 0x7f60e80b3220 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e806f230 con 0x7f60e81439c0 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ec939640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f60e80b2800 0x7f60e81432b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56270/0 (socket says 172.21.15.45:56270) 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ec939640 1 -- 172.21.15.45:0/1395371517 learned_addr learned my addr 172.21.15.45:0/1395371517 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 1 -- 172.21.15.45:0/1395371517 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e806ef50 con 0x7f60e80b2800 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60ed93b640 10 monclient(hunting): authenticate will time out at 989.952195s 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693760104 0 0) 0x7f60e806ef50 con 0x7f60e80b2800 2024-03-29T10:55:15.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60e77fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:55:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.683+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d8003520 con 0x7f60e80b2800 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076766505 0 0) 0x7f60e80b3840 con 0x7f60e80b3220 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e806ef50 con 0x7f60e80b3220 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961174561 0 0) 0x7f60e806f230 con 0x7f60e81439c0 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e80b3840 con 0x7f60e81439c0 2024-03-29T10:55:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766271929 0 0) 0x7f60d8003520 con 0x7f60e80b2800 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 >> v1:172.21.15.116:6789/0 conn(0x7f60e81439c0 legacy=0x7f60e8151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 >> v1:172.21.15.45:6789/0 conn(0x7f60e80b3220 legacy=0x7f60e8146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 monclient: found mon.noname-b 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e8071d30 con 0x7f60e80b2800 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f60dc001610 con 0x7f60e80b2800 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60e77fe640 10 monclient: dump: 2024-03-29T10:55:15.667 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60ed93b640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.684+0000 7f60ed93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60dc001770 con 0x7f60e80b2800 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60e77fe640 1 -- 172.21.15.45:0/1395371517 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f60dc001bc0 con 0x7f60e80b2800 2024-03-29T10:55:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60ed93b640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60ed93b640 4 set_mon_vals no callback set 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60e77fe640 10 monclient: dump: 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:15.669 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.685+0000 7f60ed93b640 10 monclient: shutdown 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.686+0000 7f60ed93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.686+0000 7f60ed93b640 1 -- 172.21.15.45:0/1395371517 >> v1:172.21.15.45:6790/0 conn(0x7f60e80b2800 legacy=0x7f60e81432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.686+0000 7f60ed93b640 1 -- 172.21.15.45:0/1395371517 shutdown_connections 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.686+0000 7f60ed93b640 1 -- 172.21.15.45:0/1395371517 >> 172.21.15.45:0/1395371517 conn(0x7f60e8066a00 msgr2=0x7f60e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.686+0000 7f60ed93b640 1 -- 172.21.15.45:0/1395371517 shutdown_connections 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.686+0000 7f60ed93b640 1 -- 172.21.15.45:0/1395371517 wait complete. 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.687+0000 7f60ed93b640 10 monclient: build_initial_monmap 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.687+0000 7f60ed93b640 10 monclient: monmap: 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:15.675 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 1 librados: starting msgr at 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 1 librados: starting objecter 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 1 Processor -- start 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 1 -- start start 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 1 librados: setting wanted keys 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 1 librados: calling monclient init 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.688+0000 7f60ed93b640 10 monclient: init 2024-03-29T10:55:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ed93b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ed93b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ed93b640 10 monclient(hunting): picked mon.noname-b con 0x7f60e80b2800 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ed93b640 10 monclient(hunting): picked mon.noname-c con 0x7f60e80b3220 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ec939640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f60e80b2800 0x7f60e8138f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:52488/0 (socket says 172.21.15.45:52488) 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ec939640 1 -- 172.21.15.45:0/2948277599 learned_addr learned my addr 172.21.15.45:0/2948277599 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60e7fff640 1 --1- 172.21.15.45:0/2948277599 >> v1:172.21.15.45:6790/0 conn(0x7f60e80b3220 0x7f60e81dbed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56532/0 (socket says 172.21.15.45:56532) 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ed93b640 10 monclient(hunting): picked mon.noname-a con 0x7f60e81439c0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.690+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e8147f60 con 0x7f60e81439c0 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 client.?.objecter ms_handle_connect 0x7f60e80b2800 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 client.?.objecter ms_handle_connect 0x7f60e80b3220 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e813f0c0 con 0x7f60e80b2800 2024-03-29T10:55:15.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 client.?.objecter ms_handle_connect 0x7f60e81439c0 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e813de20 con 0x7f60e80b3220 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60ed93b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60ed93b640 10 monclient(hunting): authenticate will time out at 989.960151s 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498422136 0 0) 0x7f60e813f0c0 con 0x7f60e80b2800 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.691+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60cc004eb0 con 0x7f60e80b2800 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162457446 0 0) 0x7f60e8147f60 con 0x7f60e81439c0 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:15.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e813f0c0 con 0x7f60e81439c0 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895580487 0 0) 0x7f60e813de20 con 0x7f60e80b3220 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e8147f60 con 0x7f60e80b3220 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3220026287 0 0) 0x7f60cc004eb0 con 0x7f60e80b2800 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60e813de20 con 0x7f60e80b2800 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f60dc001b40 con 0x7f60e80b2800 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): dump: 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.692+0000 7f60e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4086026171 0 0) 0x7f60e813de20 con 0x7f60e80b2800 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 >> v1:172.21.15.45:6790/0 conn(0x7f60e80b3220 legacy=0x7f60e81dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 >> v1:172.21.15.45:6789/0 conn(0x7f60e81439c0 legacy=0x7f60e81dc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 1 monclient: found mon.b 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e813dfb0 con 0x7f60e80b2800 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 10 monclient: _finish_auth 0 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 10 monclient: _renew_subs 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e80b6630 con 0x7f60e80b2800 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 10 monclient: _renew_subs 2024-03-29T10:55:15.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e8139690 con 0x7f60e80b2800 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.693+0000 7f60ed93b640 1 librados: init done 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60dc004a90 con 0x7f60e80b2800 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f60dc004da0 con 0x7f60e80b2800 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e57fa640 10 monclient: dump: 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:15.682 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:15.683 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:15.683 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:15.683 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:15.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60e6ffd640 4 set_mon_vals no callback set 2024-03-29T10:55:15.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.694+0000 7f60ed93b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:15.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.695+0000 7f60ed93b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:15.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.695+0000 7f60ed93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:15.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.695+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8005080 con 0x7f60e80b2800 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.697+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f60dc01ad70 con 0x7f60e80b2800 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f60dc084b40 con 0x7f60e80b2800 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e57fa640 10 client.4360.objecter ms_dispatch 0x7f60e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e57fa640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e57fa640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e7fff640 10 monclient: get_auth_request con 0x7f60cc06de80 auth_method 0 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e57fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.698+0000 7f60e57fa640 10 client.4360.objecter ms_handle_connect 0x7f60cc06de80 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.704+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60dc050420 con 0x7f60e80b2800 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.704+0000 7f60e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:15.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:15.704+0000 7f60e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:16.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.128+0000 7f60ed93b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.128+0000 7f60ed93b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.128+0000 7f60ed93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:16.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.128+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c8005330 con 0x7f60e80b2800 2024-03-29T10:55:16.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.130+0000 7f60e57fa640 1 -- 172.21.15.45:0/2948277599 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f60dc052350 con 0x7f60e80b2800 2024-03-29T10:55:16.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.130+0000 7f60e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.130+0000 7f60e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:16.112 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:16.112 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:16.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.133+0000 7f60ed93b640 10 librados: watch_flush enter 2024-03-29T10:55:16.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.133+0000 7f60ed93b640 10 librados: watch_flush exit 2024-03-29T10:55:16.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.133+0000 7f60ed93b640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T10:55:16.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.133+0000 7f60ed93b640 10 client.4360.objecter successfully canceled tick 2024-03-29T10:55:16.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.134+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 >> v1:172.21.15.116:6816/3290579615 conn(0x7f60cc06de80 legacy=0x7f60cc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.134+0000 7f60ed93b640 10 monclient: shutdown 2024-03-29T10:55:16.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.134+0000 7f60ed93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:16.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.134+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 >> v1:172.21.15.116:6789/0 conn(0x7f60e80b2800 legacy=0x7f60e8138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.134+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 shutdown_connections 2024-03-29T10:55:16.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.134+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 >> 172.21.15.45:0/2948277599 conn(0x7f60e8066a00 msgr2=0x7f60e8141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:16.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.135+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 shutdown_connections 2024-03-29T10:55:16.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.135+0000 7f60ed93b640 1 -- 172.21.15.45:0/2948277599 wait complete. 2024-03-29T10:55:16.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:16.135+0000 7f60ed93b640 1 librados: shutdown 2024-03-29T10:55:16.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:22.941 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.113+0000 7f5c14efb640 10 monclient: get_monmap_and_config 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.113+0000 7f5c14efb640 10 monclient: build_initial_monmap 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.113+0000 7f5c14efb640 10 monclient: monmap: 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:23.115196+0000 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:23.115196+0000 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:23.095 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:23.096 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:23.096 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:23.096 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:23.096 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:23.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.114+0000 7f5c14efb640 1 Processor -- start 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.114+0000 7f5c14efb640 1 -- start start 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.114+0000 7f5c14efb640 10 monclient: init 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c080389e0 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c08038dc0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c08030660 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c080c54c0 con 0x7f5c08038dc0 2024-03-29T10:55:23.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08036430 con 0x7f5c08030660 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08039840 con 0x7f5c080389e0 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c14efb640 10 monclient(hunting): authenticate will time out at 997.385627s 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c0f7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f5c080389e0 0x7f5c08033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37244/0 (socket says 172.21.15.45:37244) 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.115+0000 7f5c0f7fe640 1 -- 172.21.15.45:0/2354248352 learned_addr learned my addr 172.21.15.45:0/2354248352 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918482507 0 0) 0x7f5c080c54c0 con 0x7f5c08038dc0 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c00003520 con 0x7f5c08038dc0 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243364470 0 0) 0x7f5c08039840 con 0x7f5c080389e0 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c080c54c0 con 0x7f5c080389e0 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.116+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202539355 0 0) 0x7f5c080c54c0 con 0x7f5c080389e0 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 >> v1:172.21.15.116:6789/0 conn(0x7f5c08030660 legacy=0x7f5c080d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 >> v1:172.21.15.45:6789/0 conn(0x7f5c08038dc0 legacy=0x7f5c080cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08035690 con 0x7f5c080389e0 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:23.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5c04001690 con 0x7f5c080389e0 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: dump: 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:23.102 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c14efb640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c14efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c040017f0 con 0x7f5c080389e0 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c0e7fc640 1 -- 172.21.15.45:0/2354248352 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5c04001c40 con 0x7f5c080389e0 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c14efb640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:23.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.117+0000 7f5c14efb640 4 set_mon_vals no callback set 2024-03-29T10:55:23.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:23.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c0e7fc640 10 monclient: dump: 2024-03-29T10:55:23.111 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:23.111 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 10 monclient: shutdown 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 1 -- 172.21.15.45:0/2354248352 >> v1:172.21.15.45:6790/0 conn(0x7f5c080389e0 legacy=0x7f5c08033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 1 -- 172.21.15.45:0/2354248352 shutdown_connections 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 1 -- 172.21.15.45:0/2354248352 >> 172.21.15.45:0/2354248352 conn(0x7f5c08005c00 msgr2=0x7f5c08006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 1 -- 172.21.15.45:0/2354248352 shutdown_connections 2024-03-29T10:55:23.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.118+0000 7f5c14efb640 1 -- 172.21.15.45:0/2354248352 wait complete. 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.119+0000 7f5c14efb640 10 monclient: build_initial_monmap 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.119+0000 7f5c14efb640 10 monclient: monmap: 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 1 librados: starting msgr at 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 1 librados: starting objecter 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 1 Processor -- start 2024-03-29T10:55:23.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 1 -- start start 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 1 librados: setting wanted keys 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 1 librados: calling monclient init 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.120+0000 7f5c14efb640 10 monclient: init 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c08030660 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c080389e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c080c8130 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0815ff30 con 0x7f5c080389e0 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c080c7ea0 con 0x7f5c08030660 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c080c8600 con 0x7f5c080c8130 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c14efb640 10 monclient(hunting): authenticate will time out at 997.391737s 2024-03-29T10:55:23.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c0effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5c080389e0 0x7f5c08160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44008/0 (socket says 172.21.15.45:44008) 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5c0effd640 1 -- 172.21.15.45:0/1956698516 learned_addr learned my addr 172.21.15.45:0/1956698516 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.121+0000 7f5bfbfff640 10 client.?.objecter ms_handle_connect 0x7f5c080389e0 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791696777 0 0) 0x7f5c0815ff30 con 0x7f5c080389e0 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf4004eb0 con 0x7f5c080389e0 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 client.?.objecter ms_handle_connect 0x7f5c080c8130 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805735390 0 0) 0x7f5c080c8600 con 0x7f5c080c8130 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0815ff30 con 0x7f5c080c8130 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792903073 0 0) 0x7f5bf4004eb0 con 0x7f5c080389e0 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c080c8600 con 0x7f5c080389e0 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5c1004fdd0 con 0x7f5c080389e0 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.122+0000 7f5bfbfff640 10 monclient(hunting): dump: 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:23.116 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 client.?.objecter ms_handle_connect 0x7f5c08030660 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2290242789 0 0) 0x7f5c0815ff30 con 0x7f5c080c8130 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bf4004eb0 con 0x7f5c080c8130 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2882102875 0 0) 0x7f5c080c8600 con 0x7f5c080389e0 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 >> v1:172.21.15.45:6790/0 conn(0x7f5c080c8130 legacy=0x7f5c080c0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 >> v1:172.21.15.116:6789/0 conn(0x7f5c08030660 legacy=0x7f5c0802ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 monclient: found mon.a 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:23.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0815fbc0 con 0x7f5c080389e0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 monclient: _finish_auth 0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5bfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 10 monclient: _renew_subs 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c080c09a0 con 0x7f5c080389e0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 10 monclient: _renew_subs 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0802d3c0 con 0x7f5c080389e0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.123+0000 7f5c14efb640 1 librados: init done 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c1004f400 con 0x7f5c080389e0 2024-03-29T10:55:23.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bfbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5c0dffb640 4 set_mon_vals no callback set 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5c10075710 con 0x7f5c080389e0 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bfbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bfbfff640 10 monclient: dump: 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:23.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.125+0000 7f5bf9ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bf9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bf9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c08007b70 con 0x7f5c080389e0 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f5c10067bf0 con 0x7f5c080389e0 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5c0f7fe640 10 monclient: get_auth_request con 0x7f5bf406db70 auth_method 0 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f5c100ff470 con 0x7f5c080389e0 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bfbfff640 10 client.4347.objecter ms_dispatch 0x7f5c10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bfbfff640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.126+0000 7f5bfbfff640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.127+0000 7f5bfbfff640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T10:55:23.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.127+0000 7f5bfbfff640 10 client.4347.objecter ms_handle_connect 0x7f5bf406db70 2024-03-29T10:55:23.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.134+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c100cadb0 con 0x7f5c080389e0 2024-03-29T10:55:23.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.134+0000 7f5bfbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.134+0000 7f5bfbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:23.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.513+0000 7f5bf9ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:23.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.513+0000 7f5bf9ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:23.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.513+0000 7f5bf9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:23.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.513+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c08005120 con 0x7f5c080389e0 2024-03-29T10:55:23.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.514+0000 7f5bfbfff640 1 -- 172.21.15.45:0/1956698516 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f5c100cadb0 con 0x7f5c080389e0 2024-03-29T10:55:23.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.514+0000 7f5bfbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:23.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.514+0000 7f5bfbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:23.496 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:23.497 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 10 librados: watch_flush enter 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 10 librados: watch_flush exit 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 10 client.4347.objecter successfully canceled tick 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 >> v1:172.21.15.116:6816/3290579615 conn(0x7f5bf406db70 legacy=0x7f5bf4070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 10 monclient: shutdown 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:23.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 >> v1:172.21.15.45:6789/0 conn(0x7f5c080389e0 legacy=0x7f5c08160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 shutdown_connections 2024-03-29T10:55:23.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.518+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 >> 172.21.15.45:0/1956698516 conn(0x7f5c08005c00 msgr2=0x7f5c08005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:23.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.519+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 shutdown_connections 2024-03-29T10:55:23.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.519+0000 7f5bf9ffb640 1 -- 172.21.15.45:0/1956698516 wait complete. 2024-03-29T10:55:23.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:23.519+0000 7f5bf9ffb640 1 librados: shutdown 2024-03-29T10:55:23.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:30.322 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:30.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.479+0000 7f2fbea4e640 10 monclient: get_monmap_and_config 2024-03-29T10:55:30.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.479+0000 7f2fbea4e640 10 monclient: build_initial_monmap 2024-03-29T10:55:30.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.479+0000 7f2fbea4e640 10 monclient: monmap: 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:30.480596+0000 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:30.480596+0000 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:30.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.480+0000 7f2fbea4e640 1 Processor -- start 2024-03-29T10:55:30.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.480+0000 7f2fbea4e640 1 -- start start 2024-03-29T10:55:30.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.480+0000 7f2fbea4e640 10 monclient: init 2024-03-29T10:55:30.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:30.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:30.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb8070240 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:30.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb813f900 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:30.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb80ae860 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb813f400 con 0x7f2fb80ae860 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb806f760 con 0x7f2fb813f900 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb81429c0 con 0x7f2fb8070240 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbea4e640 10 monclient(hunting): authenticate will time out at 1004.751220s 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbcfeb640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2fb80ae860 0x7f2fb814b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35502/0 (socket says 172.21.15.45:35502) 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fbcfeb640 1 -- 172.21.15.45:0/270146724 learned_addr learned my addr 172.21.15.45:0/270146724 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615352068 0 0) 0x7f2fb813f400 con 0x7f2fb80ae860 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa0003520 con 0x7f2fb80ae860 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248619100 0 0) 0x7f2fb81429c0 con 0x7f2fb8070240 2024-03-29T10:55:30.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:55:30.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.482+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb813f400 con 0x7f2fb8070240 2024-03-29T10:55:30.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.483+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2021439800 0 0) 0x7f2fb813f400 con 0x7f2fb8070240 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.483+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 >> v1:172.21.15.116:6789/0 conn(0x7f2fb813f900 legacy=0x7f2fb81422b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 >> v1:172.21.15.45:6789/0 conn(0x7f2fb80ae860 legacy=0x7f2fb814b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8145610 con 0x7f2fb8070240 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2fb0001970 con 0x7f2fb8070240 2024-03-29T10:55:30.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fb6ffd640 10 monclient: dump: 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fbea4e640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.484+0000 7f2fbea4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:30.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb0001b90 con 0x7f2fb8070240 2024-03-29T10:55:30.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:30.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fbea4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:30.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fbea4e640 4 set_mon_vals no callback set 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fb6ffd640 1 -- 172.21.15.45:0/270146724 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2fb0001f20 con 0x7f2fb8070240 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.485+0000 7f2fb6ffd640 10 monclient: dump: 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.469 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.486+0000 7f2fbea4e640 10 monclient: shutdown 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.486+0000 7f2fbea4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.486+0000 7f2fbea4e640 1 -- 172.21.15.45:0/270146724 >> v1:172.21.15.45:6790/0 conn(0x7f2fb8070240 legacy=0x7f2fb806f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.486+0000 7f2fbea4e640 1 -- 172.21.15.45:0/270146724 shutdown_connections 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.486+0000 7f2fbea4e640 1 -- 172.21.15.45:0/270146724 >> 172.21.15.45:0/270146724 conn(0x7f2fb80b4930 msgr2=0x7f2fb80b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.488+0000 7f2fbea4e640 1 -- 172.21.15.45:0/270146724 shutdown_connections 2024-03-29T10:55:30.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.488+0000 7f2fbea4e640 1 -- 172.21.15.45:0/270146724 wait complete. 2024-03-29T10:55:30.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.489+0000 7f2fbea4e640 10 monclient: build_initial_monmap 2024-03-29T10:55:30.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.489+0000 7f2fbea4e640 10 monclient: monmap: 2024-03-29T10:55:30.474 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:30.474 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:30.474 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:30.474 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 1 librados: starting msgr at 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 1 librados: starting objecter 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 1 Processor -- start 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 1 -- start start 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 1 librados: setting wanted keys 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 1 librados: calling monclient init 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.490+0000 7f2fbea4e640 10 monclient: init 2024-03-29T10:55:30.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb8070240 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb80ae860 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb813f900 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb81df040 con 0x7f2fb813f900 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb81df560 con 0x7f2fb80ae860 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb813c470 con 0x7f2fb8070240 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fbea4e640 10 monclient(hunting): authenticate will time out at 1004.760376s 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fb7fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2fb8070240 0x7f2fb81de730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58840/0 (socket says 172.21.15.45:58840) 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fb7fff640 1 -- 172.21.15.45:0/1299406988 learned_addr learned my addr 172.21.15.45:0/1299406988 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f2fb813f900 2024-03-29T10:55:30.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.491+0000 7f2fb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828770249 0 0) 0x7f2fb81df040 con 0x7f2fb813f900 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa4004eb0 con 0x7f2fb813f900 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f2fb8070240 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481388139 0 0) 0x7f2fb813c470 con 0x7f2fb8070240 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb81df040 con 0x7f2fb8070240 2024-03-29T10:55:30.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2990364178 0 0) 0x7f2fa4004eb0 con 0x7f2fb813f900 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb813c470 con 0x7f2fb813f900 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2fa80018d0 con 0x7f2fb813f900 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:30.478 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:30.479 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:30.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:30.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127206803 0 0) 0x7f2fb813c470 con 0x7f2fb813f900 2024-03-29T10:55:30.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 >> v1:172.21.15.45:6790/0 conn(0x7f2fb8070240 legacy=0x7f2fb81de730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.492+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 >> v1:172.21.15.116:6789/0 conn(0x7f2fb80ae860 legacy=0x7f2fb80aabf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 1 monclient: found mon.a 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80ab300 con 0x7f2fb813f900 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 10 monclient: _renew_subs 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:30.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb80ab570 con 0x7f2fb813f900 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 10 monclient: _renew_subs 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb813e110 con 0x7f2fb813f900 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fbea4e640 1 librados: init done 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fa80047e0 con 0x7f2fb813f900 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2fa8004af0 con 0x7f2fb813f900 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb4ff9640 10 monclient: dump: 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:30.483 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.494+0000 7f2fb67fc640 4 set_mon_vals no callback set 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.495+0000 7f2fbea4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.495+0000 7f2fbea4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.495+0000 7f2fbea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.495+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f98005080 con 0x7f2fb813f900 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.497+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f2fa801aac0 con 0x7f2fb813f900 2024-03-29T10:55:30.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.498+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f2fa8084950 con 0x7f2fb813f900 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.498+0000 7f2fb4ff9640 10 client.4353.objecter ms_dispatch 0x7f2fb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.498+0000 7f2fb4ff9640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.498+0000 7f2fb4ff9640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.498+0000 7f2fb4ff9640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.498+0000 7f2fb7fff640 10 monclient: get_auth_request con 0x7f2fa406daa0 auth_method 0 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.499+0000 7f2fb4ff9640 10 client.4353.objecter ms_handle_connect 0x7f2fa406daa0 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.504+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fa8050230 con 0x7f2fb813f900 2024-03-29T10:55:30.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.504+0000 7f2fb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:30.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.504+0000 7f2fb4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:30.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fbea4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:30.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fbea4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:30.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fbea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:55:30.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f98005330 con 0x7f2fb813f900 2024-03-29T10:55:30.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fb4ff9640 1 -- 172.21.15.45:0/1299406988 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f2fa8052160 con 0x7f2fb813f900 2024-03-29T10:55:30.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:30.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.825+0000 7f2fb4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:30.807 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:30.807 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:30.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.828+0000 7f2fbea4e640 10 librados: watch_flush enter 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.829+0000 7f2fbea4e640 10 librados: watch_flush exit 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.829+0000 7f2fbea4e640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.829+0000 7f2fbea4e640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 >> v1:172.21.15.116:6816/3290579615 conn(0x7f2fa406daa0 legacy=0x7f2fa406ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 10 monclient: shutdown 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 >> v1:172.21.15.45:6789/0 conn(0x7f2fb813f900 legacy=0x7f2fb813a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 shutdown_connections 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 >> 172.21.15.45:0/1299406988 conn(0x7f2fb80b4930 msgr2=0x7f2fb80718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 shutdown_connections 2024-03-29T10:55:30.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 -- 172.21.15.45:0/1299406988 wait complete. 2024-03-29T10:55:30.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:30.830+0000 7f2fbea4e640 1 librados: shutdown 2024-03-29T10:55:30.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:37.633 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.801+0000 7f20b3025640 10 monclient: get_monmap_and_config 2024-03-29T10:55:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.801+0000 7f20b3025640 10 monclient: build_initial_monmap 2024-03-29T10:55:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.801+0000 7f20b3025640 10 monclient: monmap: 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:37.802589+0000 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:37.802589+0000 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:37.783 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.802+0000 7f20b3025640 1 Processor -- start 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.802+0000 7f20b3025640 1 -- start start 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient: init 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient(hunting): picked mon.noname-a con 0x7f20ac149c50 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient(hunting): picked mon.noname-c con 0x7f20ac0a9cc0 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient(hunting): picked mon.noname-b con 0x7f20ac143d00 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0ab9c0 con 0x7f20ac149c50 2024-03-29T10:55:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0abc10 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0aad20 con 0x7f20ac143d00 2024-03-29T10:55:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.803+0000 7f20b3025640 10 monclient(hunting): authenticate will time out at 1012.072864s 2024-03-29T10:55:37.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.807+0000 7f20b15c2640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f20ac143d00 0x7f20ac151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:52112/0 (socket says 172.21.15.45:52112) 2024-03-29T10:55:37.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.807+0000 7f20b15c2640 1 -- 172.21.15.45:0/204296269 learned_addr learned my addr 172.21.15.45:0/204296269 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:37.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533759934 0 0) 0x7f20ac0aad20 con 0x7f20ac143d00 2024-03-29T10:55:37.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2098003520 con 0x7f20ac143d00 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 738602476 0 0) 0x7f2098003520 con 0x7f20ac143d00 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 >> v1:172.21.15.116:6789/0 conn(0x7f20ac0a9cc0 legacy=0x7f20ac1435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 >> v1:172.21.15.45:6789/0 conn(0x7f20ac149c50 legacy=0x7f20ac0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 monclient: found mon.noname-b 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:55:37.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac0aa980 con 0x7f20ac143d00 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f209c001a10 con 0x7f20ac143d00 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: dump: 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:37.791 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f209c001be0 con 0x7f20ac143d00 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20b3025640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20b3025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 1 -- 172.21.15.45:0/204296269 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f209c002030 con 0x7f20ac143d00 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20b3025640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20b3025640 4 set_mon_vals no callback set 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.808+0000 7f20ab7fe640 10 monclient: dump: 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:37.792 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:37.793 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.809+0000 7f20b3025640 10 monclient: shutdown 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.809+0000 7f20b3025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.809+0000 7f20b3025640 1 -- 172.21.15.45:0/204296269 >> v1:172.21.15.45:6790/0 conn(0x7f20ac143d00 legacy=0x7f20ac151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.810+0000 7f20b3025640 1 -- 172.21.15.45:0/204296269 shutdown_connections 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.810+0000 7f20b3025640 1 -- 172.21.15.45:0/204296269 >> 172.21.15.45:0/204296269 conn(0x7f20ac0b3c50 msgr2=0x7f20ac0b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.811+0000 7f20b3025640 1 -- 172.21.15.45:0/204296269 shutdown_connections 2024-03-29T10:55:37.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.812+0000 7f20b3025640 1 -- 172.21.15.45:0/204296269 wait complete. 2024-03-29T10:55:37.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.813+0000 7f20b3025640 10 monclient: build_initial_monmap 2024-03-29T10:55:37.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.813+0000 7f20b3025640 10 monclient: monmap: 2024-03-29T10:55:37.795 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:37.795 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:37.795 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 1 librados: starting msgr at 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 1 librados: starting objecter 2024-03-29T10:55:37.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 1 Processor -- start 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 1 -- start start 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 1 librados: setting wanted keys 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 1 librados: calling monclient init 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.814+0000 7f20b3025640 10 monclient: init 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient(hunting): picked mon.noname-b con 0x7f20ac0a9cc0 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient(hunting): picked mon.noname-c con 0x7f20ac143d00 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient(hunting): picked mon.noname-a con 0x7f20ac149c50 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:37.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac071780 con 0x7f20ac149c50 2024-03-29T10:55:37.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0701e0 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac06f790 con 0x7f20ac143d00 2024-03-29T10:55:37.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:37.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.815+0000 7f20b3025640 10 monclient(hunting): authenticate will time out at 1012.084416s 2024-03-29T10:55:37.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.818+0000 7f20abfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f20ac143d00 0x7f20ac1db230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:52296/0 (socket says 172.21.15.45:52296) 2024-03-29T10:55:37.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.818+0000 7f20abfff640 1 -- 172.21.15.45:0/3494720941 learned_addr learned my addr 172.21.15.45:0/3494720941 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.820+0000 7f20a97fa640 10 client.?.objecter ms_handle_connect 0x7f20ac143d00 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.820+0000 7f20a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.820+0000 7f20a97fa640 10 client.?.objecter ms_handle_connect 0x7f20ac0a9cc0 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.820+0000 7f20a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322017444 0 0) 0x7f20ac0701e0 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:55:37.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f208c004eb0 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219323995 0 0) 0x7f20ac06f790 con 0x7f20ac143d00 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ac0701e0 con 0x7f20ac143d00 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 client.?.objecter ms_handle_connect 0x7f20ac149c50 2024-03-29T10:55:37.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:37.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483985937 0 0) 0x7f20ac071780 con 0x7f20ac149c50 2024-03-29T10:55:37.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ac06f790 con 0x7f20ac149c50 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757526487 0 0) 0x7f20ac0701e0 con 0x7f20ac143d00 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.821+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20ac071780 con 0x7f20ac143d00 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 954204897 0 0) 0x7f208c004eb0 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20ac0701e0 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f20a4004040 con 0x7f20ac143d00 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): dump: 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:37.806 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2094001680 con 0x7f20ac0a9cc0 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient(hunting): dump: 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932925199 0 0) 0x7f20ac06f790 con 0x7f20ac149c50 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f208c004eb0 con 0x7f20ac149c50 2024-03-29T10:55:37.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103658196 0 0) 0x7f20ac071780 con 0x7f20ac143d00 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 >> v1:172.21.15.116:6789/0 conn(0x7f20ac0a9cc0 legacy=0x7f20ac13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 >> v1:172.21.15.45:6789/0 conn(0x7f20ac149c50 legacy=0x7f20ac1db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 monclient: found mon.c 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac06f920 con 0x7f20ac143d00 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient: _finish_auth 0 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20a40041a0 con 0x7f20ac143d00 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f20a40010b0 con 0x7f20ac143d00 2024-03-29T10:55:37.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.822+0000 7f20aaffd640 4 set_mon_vals no callback set 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.823+0000 7f20a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20a97fa640 10 monclient: dump: 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:37.810 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 10 monclient: _renew_subs 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac06fbe0 con 0x7f20ac143d00 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 10 monclient: _renew_subs 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac146040 con 0x7f20ac143d00 2024-03-29T10:55:37.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.824+0000 7f20b3025640 1 librados: init done 2024-03-29T10:55:37.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.826+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f20a4001290 con 0x7f20ac143d00 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.826+0000 7f20b3025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20b3025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f20a40849e0 con 0x7f20ac143d00 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20a97fa640 10 client.4355.objecter ms_dispatch 0x7f20ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20a97fa640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20a97fa640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20a97fa640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20b3025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.827+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac070790 con 0x7f20ac143d00 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.828+0000 7f20b0dc1640 10 monclient: get_auth_request con 0x7f208c06e170 auth_method 0 2024-03-29T10:55:37.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.830+0000 7f20a97fa640 10 client.4355.objecter ms_handle_connect 0x7f208c06e170 2024-03-29T10:55:37.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.838+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20a40500f0 con 0x7f20ac143d00 2024-03-29T10:55:37.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.838+0000 7f20a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:37.838+0000 7f20a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:38.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20b3025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:38.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20b3025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:38.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20b3025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:38.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20ac070370 con 0x7f20ac143d00 2024-03-29T10:55:38.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20a97fa640 1 -- 172.21.15.45:0/3494720941 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f20a40521f0 con 0x7f20ac143d00 2024-03-29T10:55:38.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:38.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.184+0000 7f20a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:38.166 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:38.166 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:38.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.188+0000 7f20b3025640 10 librados: watch_flush enter 2024-03-29T10:55:38.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.188+0000 7f20b3025640 10 librados: watch_flush exit 2024-03-29T10:55:38.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.188+0000 7f20b3025640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T10:55:38.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.188+0000 7f20b3025640 10 client.4355.objecter successfully canceled tick 2024-03-29T10:55:38.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 >> v1:172.21.15.116:6816/3290579615 conn(0x7f208c06e170 legacy=0x7f208c070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 10 monclient: shutdown 2024-03-29T10:55:38.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:38.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 >> v1:172.21.15.45:6790/0 conn(0x7f20ac143d00 legacy=0x7f20ac1db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 shutdown_connections 2024-03-29T10:55:38.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 >> 172.21.15.45:0/3494720941 conn(0x7f20ac0b3c50 msgr2=0x7f20ac142b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:38.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 shutdown_connections 2024-03-29T10:55:38.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 -- 172.21.15.45:0/3494720941 wait complete. 2024-03-29T10:55:38.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:38.189+0000 7f20b3025640 1 librados: shutdown 2024-03-29T10:55:38.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:44.995 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:45.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.149+0000 7faf52095640 10 monclient: get_monmap_and_config 2024-03-29T10:55:45.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.149+0000 7faf52095640 10 monclient: build_initial_monmap 2024-03-29T10:55:45.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.149+0000 7faf52095640 10 monclient: monmap: 2024-03-29T10:55:45.130 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:45.150397+0000 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:45.150397+0000 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:45.131 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.150+0000 7faf52095640 1 Processor -- start 2024-03-29T10:55:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.150+0000 7faf52095640 1 -- start start 2024-03-29T10:55:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.150+0000 7faf52095640 10 monclient: init 2024-03-29T10:55:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient(hunting): picked mon.noname-a con 0x7faf4c13f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient(hunting): picked mon.noname-b con 0x7faf4c0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0aaef0 con 0x7faf4c13f650 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c145bd0 con 0x7faf4c1457f0 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c145eb0 con 0x7faf4c0a9d70 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.151+0000 7faf52095640 10 monclient(hunting): authenticate will time out at 1019.420833s 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf4c1457f0 0x7faf4c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:44588/0 (socket says 172.21.15.45:44588) 2024-03-29T10:55:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4b7fe640 1 -- 172.21.15.45:0/3665872212 learned_addr learned my addr 172.21.15.45:0/3665872212 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658755126 0 0) 0x7faf4c145bd0 con 0x7faf4c1457f0 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf34003520 con 0x7faf4c1457f0 2024-03-29T10:55:45.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102619626 0 0) 0x7faf34003520 con 0x7faf4c1457f0 2024-03-29T10:55:45.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.152+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 >> v1:172.21.15.45:6790/0 conn(0x7faf4c0a9d70 legacy=0x7faf4c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:45.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.153+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 >> v1:172.21.15.45:6789/0 conn(0x7faf4c13f650 legacy=0x7faf4c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c146340 con 0x7faf4c1457f0 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:45.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf52095640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faf3c001690 con 0x7faf4c1457f0 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf52095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf4a7fc640 10 monclient: dump: 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:45.137 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:45.138 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:45.138 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:45.138 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:45.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.155+0000 7faf52095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf3c0019a0 con 0x7faf4c1457f0 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf4a7fc640 1 -- 172.21.15.45:0/3665872212 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faf3c001d30 con 0x7faf4c1457f0 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf4a7fc640 10 monclient: dump: 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:45.140 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf52095640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.156+0000 7faf52095640 4 set_mon_vals no callback set 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 10 monclient: shutdown 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 1 -- 172.21.15.45:0/3665872212 >> v1:172.21.15.116:6789/0 conn(0x7faf4c1457f0 legacy=0x7faf4c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 1 -- 172.21.15.45:0/3665872212 shutdown_connections 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 1 -- 172.21.15.45:0/3665872212 >> 172.21.15.45:0/3665872212 conn(0x7faf4c066a00 msgr2=0x7faf4c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 1 -- 172.21.15.45:0/3665872212 shutdown_connections 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.157+0000 7faf52095640 1 -- 172.21.15.45:0/3665872212 wait complete. 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.158+0000 7faf52095640 10 monclient: build_initial_monmap 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.158+0000 7faf52095640 10 monclient: monmap: 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:45.142 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.158+0000 7faf52095640 1 librados: starting msgr at 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.158+0000 7faf52095640 1 librados: starting objecter 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.158+0000 7faf52095640 1 Processor -- start 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 1 -- start start 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 1 librados: setting wanted keys 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 1 librados: calling monclient init 2024-03-29T10:55:45.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 10 monclient: init 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.159+0000 7faf52095640 10 monclient(hunting): picked mon.noname-b con 0x7faf4c0a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c13f650 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf4b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf4c0a9d70 0x7faf4c1d7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:45466/0 (socket says 172.21.15.45:45466) 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 10 monclient(hunting): picked mon.noname-a con 0x7faf4c1457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c135be0 con 0x7faf4c1457f0 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c133fe0 con 0x7faf4c0a9d70 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c134ba0 con 0x7faf4c13f650 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:45.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf52095640 10 monclient(hunting): authenticate will time out at 1019.429206s 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf4affd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7faf4c13f650 0x7faf4c130cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56424/0 (socket says 172.21.15.45:56424) 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf4affd640 1 -- 172.21.15.45:0/1780771997 learned_addr learned my addr 172.21.15.45:0/1780771997 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf33fff640 10 client.?.objecter ms_handle_connect 0x7faf4c13f650 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.160+0000 7faf33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525462852 0 0) 0x7faf4c134ba0 con 0x7faf4c13f650 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf38004eb0 con 0x7faf4c13f650 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 client.?.objecter ms_handle_connect 0x7faf4c0a9d70 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074786847 0 0) 0x7faf38004eb0 con 0x7faf4c13f650 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf4c134ba0 con 0x7faf4c13f650 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faf40001960 con 0x7faf4c13f650 2024-03-29T10:55:45.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): dump: 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878175554 0 0) 0x7faf4c133fe0 con 0x7faf4c0a9d70 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:45.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf38004eb0 con 0x7faf4c0a9d70 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 183922104 0 0) 0x7faf4c134ba0 con 0x7faf4c13f650 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 >> v1:172.21.15.116:6789/0 conn(0x7faf4c0a9d70 legacy=0x7faf4c1d7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.161+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 >> v1:172.21.15.45:6789/0 conn(0x7faf4c1457f0 legacy=0x7faf4c1e2ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 1 monclient: found mon.c 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c134d30 con 0x7faf4c13f650 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 5 monclient: authenticate success, global_id 4358 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 10 monclient: _renew_subs 2024-03-29T10:55:45.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c1313e0 con 0x7faf4c13f650 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 10 monclient: _renew_subs 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c142700 con 0x7faf4c13f650 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf52095640 1 librados: init done 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf40001c40 con 0x7faf4c13f650 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faf40001ac0 con 0x7faf4c13f650 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf33fff640 10 monclient: dump: 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:45.154 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.162+0000 7faf49ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.164+0000 7faf52095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.164+0000 7faf52095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.164+0000 7faf52095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.164+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c142a50 con 0x7faf4c13f650 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.165+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7faf40003a40 con 0x7faf4c13f650 2024-03-29T10:55:45.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.166+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7faf40084ab0 con 0x7faf4c13f650 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.166+0000 7faf33fff640 10 client.4358.objecter ms_dispatch 0x7faf4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.166+0000 7faf33fff640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.166+0000 7faf33fff640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.166+0000 7faf33fff640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.166+0000 7faf4b7fe640 10 monclient: get_auth_request con 0x7faf3806deb0 auth_method 0 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.168+0000 7faf33fff640 10 client.4358.objecter ms_handle_connect 0x7faf3806deb0 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.172+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf40050390 con 0x7faf4c13f650 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.172+0000 7faf33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:45.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.172+0000 7faf33fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:45.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.493+0000 7faf52095640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:45.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.494+0000 7faf52095640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:45.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.494+0000 7faf52095640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:45.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.494+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf4c075f10 con 0x7faf4c13f650 2024-03-29T10:55:45.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.496+0000 7faf33fff640 1 -- 172.21.15.45:0/1780771997 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7faf400522c0 con 0x7faf4c13f650 2024-03-29T10:55:45.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.496+0000 7faf33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:45.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.496+0000 7faf33fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:45.479 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:45.479 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 10 librados: watch_flush enter 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 10 librados: watch_flush exit 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 10 client.4358.objecter successfully canceled tick 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 >> v1:172.21.15.116:6816/3290579615 conn(0x7faf3806deb0 legacy=0x7faf38070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 10 monclient: shutdown 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 >> v1:172.21.15.45:6790/0 conn(0x7faf4c13f650 legacy=0x7faf4c130cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 shutdown_connections 2024-03-29T10:55:45.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.500+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 >> 172.21.15.45:0/1780771997 conn(0x7faf4c066a00 msgr2=0x7faf4c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:45.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.501+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 shutdown_connections 2024-03-29T10:55:45.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.501+0000 7faf52095640 1 -- 172.21.15.45:0/1780771997 wait complete. 2024-03-29T10:55:45.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:45.501+0000 7faf52095640 1 librados: shutdown 2024-03-29T10:55:45.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:52.305 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:52.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.459+0000 7f019a686640 10 monclient: get_monmap_and_config 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.459+0000 7f019a686640 10 monclient: build_initial_monmap 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.459+0000 7f019a686640 10 monclient: monmap: 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:52.460155+0000 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:52.460155+0000 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:52.440 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:52.441 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:52.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 1 Processor -- start 2024-03-29T10:55:52.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 1 -- start start 2024-03-29T10:55:52.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 10 monclient: init 2024-03-29T10:55:52.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:52.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 10 monclient(hunting): picked mon.noname-c con 0x7f01940b4790 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.460+0000 7f019a686640 10 monclient(hunting): picked mon.noname-b con 0x7f01940aea90 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f019a686640 10 monclient(hunting): picked mon.noname-a con 0x7f01940b39f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f019a686640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01940b0c60 con 0x7f01940b39f0 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f019a686640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01940af020 con 0x7f01940b4790 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f019a686640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019413fbd0 con 0x7f01940aea90 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f019a686640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f019a686640 10 monclient(hunting): authenticate will time out at 1026.730086s 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0193fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f01940b4790 0x7f01940b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:59864/0 (socket says 172.21.15.45:59864) 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0193fff640 1 -- 172.21.15.45:0/3663966412 learned_addr learned my addr 172.21.15.45:0/3663966412 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025267936 0 0) 0x7f01940af020 con 0x7f01940b4790 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:52.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:52.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:52.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:55:52.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017c003520 con 0x7f01940b4790 2024-03-29T10:55:52.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 231795314 0 0) 0x7f017c003520 con 0x7f01940b4790 2024-03-29T10:55:52.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 >> v1:172.21.15.45:6790/0 conn(0x7f01940aea90 legacy=0x7f01940af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.461+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 >> v1:172.21.15.45:6789/0 conn(0x7f01940b39f0 legacy=0x7f019413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 1 monclient: found mon.noname-c 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019413fd60 con 0x7f01940b4790 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f019a686640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0184001690 con 0x7f01940b4790 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: dump: 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f019a686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0184001860 con 0x7f01940b4790 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:52.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 1 -- 172.21.15.45:0/3663966412 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0184001cb0 con 0x7f01940b4790 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f0192ffd640 10 monclient: dump: 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:52.449 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f019a686640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.464+0000 7f019a686640 4 set_mon_vals no callback set 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.465+0000 7f019a686640 10 monclient: shutdown 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.465+0000 7f019a686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.465+0000 7f019a686640 1 -- 172.21.15.45:0/3663966412 >> v1:172.21.15.116:6789/0 conn(0x7f01940b4790 legacy=0x7f01940b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.465+0000 7f019a686640 1 -- 172.21.15.45:0/3663966412 shutdown_connections 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.465+0000 7f019a686640 1 -- 172.21.15.45:0/3663966412 >> 172.21.15.45:0/3663966412 conn(0x7f0194066a00 msgr2=0x7f0194066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.466+0000 7f019a686640 1 -- 172.21.15.45:0/3663966412 shutdown_connections 2024-03-29T10:55:52.450 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.466+0000 7f019a686640 1 -- 172.21.15.45:0/3663966412 wait complete. 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.467+0000 7f019a686640 10 monclient: build_initial_monmap 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.467+0000 7f019a686640 10 monclient: monmap: 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:52.454 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.467+0000 7f019a686640 1 librados: starting msgr at 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.467+0000 7f019a686640 1 librados: starting objecter 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.467+0000 7f019a686640 1 Processor -- start 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 1 -- start start 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 1 librados: setting wanted keys 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 1 librados: calling monclient init 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 10 monclient: init 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:52.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 10 monclient(hunting): picked mon.noname-c con 0x7f01940b39f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.468+0000 7f019a686640 10 monclient(hunting): picked mon.noname-b con 0x7f01940b4790 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f019a686640 10 monclient(hunting): picked mon.noname-a con 0x7f01940a9e30 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f019a686640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01940aa5a0 con 0x7f01940a9e30 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f019a686640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01940ab690 con 0x7f01940b4790 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f019a686640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01940ab970 con 0x7f01940b39f0 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f019a686640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f019a686640 10 monclient(hunting): authenticate will time out at 1026.738796s 2024-03-29T10:55:52.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f01937fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f01940b4790 0x7f01941334e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:60594/0 (socket says 172.21.15.45:60594) 2024-03-29T10:55:52.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.469+0000 7f01937fe640 1 -- 172.21.15.45:0/3591806743 learned_addr learned my addr 172.21.15.45:0/3591806743 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:52.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 client.?.objecter ms_handle_connect 0x7f01940b4790 2024-03-29T10:55:52.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:52.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 client.?.objecter ms_handle_connect 0x7f01940b39f0 2024-03-29T10:55:52.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:52.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381413068 0 0) 0x7f01940ab970 con 0x7f01940b39f0 2024-03-29T10:55:52.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:52.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:52.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:52.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:55:52.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180004eb0 con 0x7f01940b39f0 2024-03-29T10:55:52.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361403556 0 0) 0x7f01940ab690 con 0x7f01940b4790 2024-03-29T10:55:52.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:52.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:52.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:52.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01940ab970 con 0x7f01940b4790 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1938763022 0 0) 0x7f01940ab970 con 0x7f01940b4790 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01940ab690 con 0x7f01940b4790 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2612942350 0 0) 0x7f0180004eb0 con 0x7f01940b39f0 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0180006950 con 0x7f01940b39f0 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0184001b70 con 0x7f01940b39f0 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.460 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.470+0000 7f0190ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.471+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3220052966 0 0) 0x7f0180006950 con 0x7f01940b39f0 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.471+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 >> v1:172.21.15.116:6789/0 conn(0x7f01940b4790 legacy=0x7f01941334e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.471+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 >> v1:172.21.15.45:6789/0 conn(0x7f01940a9e30 legacy=0x7f0194133bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 1 monclient: found mon.c 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:52.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940aa2a0 con 0x7f01940b39f0 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f0190ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 10 monclient: _renew_subs 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01940a9750 con 0x7f01940b39f0 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 10 monclient: _renew_subs 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0194130260 con 0x7f01940b39f0 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.472+0000 7f019a686640 1 librados: init done 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.473+0000 7f017a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:52.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.473+0000 7f017a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:52.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.473+0000 7f017a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:52.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.473+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174005080 con 0x7f01940b39f0 2024-03-29T10:55:52.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0184001d10 con 0x7f01940b39f0 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f0190ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0184004c30 con 0x7f01940b39f0 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f0190ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f0190ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f0190ff9640 10 monclient: dump: 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:52.464 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.474+0000 7f01927fc640 4 set_mon_vals no callback set 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f018401ac00 con 0x7f01940b39f0 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f01840849b0 con 0x7f01940b39f0 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f0190ff9640 10 client.4364.objecter ms_dispatch 0x7f0194000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f0190ff9640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f0190ff9640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f0190ff9640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.475+0000 7f01937fe640 10 monclient: get_auth_request con 0x7f018006e040 auth_method 0 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.476+0000 7f0190ff9640 10 client.4364.objecter ms_handle_connect 0x7f018006e040 2024-03-29T10:55:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.481+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0184050290 con 0x7f01940b39f0 2024-03-29T10:55:52.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.481+0000 7f0190ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:52.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.481+0000 7f0190ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:52.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.786+0000 7f017a7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:52.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.786+0000 7f017a7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:52.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.786+0000 7f017a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:55:52.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.786+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0174005330 con 0x7f01940b39f0 2024-03-29T10:55:52.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.787+0000 7f0190ff9640 1 -- 172.21.15.45:0/3591806743 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f01840521c0 con 0x7f01940b39f0 2024-03-29T10:55:52.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.787+0000 7f0190ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:52.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.787+0000 7f0190ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:52.768 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:55:52.768 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:52.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 10 librados: watch_flush enter 2024-03-29T10:55:52.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 10 librados: watch_flush exit 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 10 client.4364.objecter successfully canceled tick 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 >> v1:172.21.15.116:6816/3290579615 conn(0x7f018006e040 legacy=0x7f0180070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 10 monclient: shutdown 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 >> v1:172.21.15.45:6790/0 conn(0x7f01940b39f0 legacy=0x7f019412fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 shutdown_connections 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.789+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 >> 172.21.15.45:0/3591806743 conn(0x7f0194066a00 msgr2=0x7f01940b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.790+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 shutdown_connections 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.790+0000 7f017a7fc640 1 -- 172.21.15.45:0/3591806743 wait complete. 2024-03-29T10:55:52.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:52.790+0000 7f017a7fc640 1 librados: shutdown 2024-03-29T10:55:52.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:59.592 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:59.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.754+0000 7f64c921e640 10 monclient: get_monmap_and_config 2024-03-29T10:55:59.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.754+0000 7f64c921e640 10 monclient: build_initial_monmap 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.754+0000 7f64c921e640 10 monclient: monmap: 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:55:59.755799+0000 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:55:59.755799+0000 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:55:59.736 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:59.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.757+0000 7f64c921e640 1 Processor -- start 2024-03-29T10:55:59.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.758+0000 7f64c921e640 1 -- start start 2024-03-29T10:55:59.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.758+0000 7f64c921e640 10 monclient: init 2024-03-29T10:55:59.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient(hunting): picked mon.noname-b con 0x7f64c41457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient(hunting): picked mon.noname-c con 0x7f64c413f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient(hunting): picked mon.noname-a con 0x7f64c40a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c40aaef0 con 0x7f64c40a9d70 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c4145bd0 con 0x7f64c413f650 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c4145eb0 con 0x7f64c41457f0 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.759+0000 7f64c921e640 10 monclient(hunting): authenticate will time out at 1034.028399s 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.760+0000 7f64c259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f64c413f650 0x7f64c40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:51830/0 (socket says 172.21.15.45:51830) 2024-03-29T10:55:59.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.760+0000 7f64c259c640 1 -- 172.21.15.45:0/434454473 learned_addr learned my addr 172.21.15.45:0/434454473 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598799924 0 0) 0x7f64c4145bd0 con 0x7f64c413f650 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac003520 con 0x7f64c413f650 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236381254 0 0) 0x7f64c40aaef0 con 0x7f64c40a9d70 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c4145bd0 con 0x7f64c40a9d70 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123402504 0 0) 0x7f64c4145eb0 con 0x7f64c41457f0 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.763+0000 7f64c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c40aaef0 con 0x7f64c41457f0 2024-03-29T10:55:59.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4177765529 0 0) 0x7f64c4145bd0 con 0x7f64c40a9d70 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 >> v1:172.21.15.45:6790/0 conn(0x7f64c41457f0 legacy=0x7f64c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 >> v1:172.21.15.116:6789/0 conn(0x7f64c413f650 legacy=0x7f64c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 monclient: found mon.noname-a 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4146340 con 0x7f64c40a9d70 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: _finish_auth 0 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f64bc0019f0 con 0x7f64c40a9d70 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c921e640 5 monclient: authenticate success, global_id 4359 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c921e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:59.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: dump: 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c921e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64bc001b50 con 0x7f64c40a9d70 2024-03-29T10:55:59.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 1 -- 172.21.15.45:0/434454473 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f64bc001f60 con 0x7f64c40a9d70 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c1d9b640 10 monclient: dump: 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:59.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c921e640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:59.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.764+0000 7f64c921e640 4 set_mon_vals no callback set 2024-03-29T10:55:59.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.765+0000 7f64c921e640 10 monclient: shutdown 2024-03-29T10:55:59.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.765+0000 7f64c921e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:59.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.765+0000 7f64c921e640 1 -- 172.21.15.45:0/434454473 >> v1:172.21.15.45:6789/0 conn(0x7f64c40a9d70 legacy=0x7f64c414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.765+0000 7f64c921e640 1 -- 172.21.15.45:0/434454473 shutdown_connections 2024-03-29T10:55:59.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.765+0000 7f64c921e640 1 -- 172.21.15.45:0/434454473 >> 172.21.15.45:0/434454473 conn(0x7f64c4066a00 msgr2=0x7f64c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.766+0000 7f64c921e640 1 -- 172.21.15.45:0/434454473 shutdown_connections 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.766+0000 7f64c921e640 1 -- 172.21.15.45:0/434454473 wait complete. 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.767+0000 7f64c921e640 10 monclient: build_initial_monmap 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.767+0000 7f64c921e640 10 monclient: monmap: 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:59.751 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 1 librados: starting msgr at 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 1 librados: starting objecter 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 1 Processor -- start 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 1 -- start start 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 1 librados: setting wanted keys 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 1 librados: calling monclient init 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.768+0000 7f64c921e640 10 monclient: init 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient(hunting): picked mon.noname-c con 0x7f64c40a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:55:59.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient(hunting): picked mon.noname-b con 0x7f64c413f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient(hunting): picked mon.noname-a con 0x7f64c41457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c413c3d0 con 0x7f64c41457f0 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c413c860 con 0x7f64c413f650 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c40b1750 con 0x7f64c40a9d70 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.769+0000 7f64c921e640 10 monclient(hunting): authenticate will time out at 1034.038265s 2024-03-29T10:55:59.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64c259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f64c413f650 0x7f64c406f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:54006/0 (socket says 172.21.15.45:54006) 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64c259c640 1 -- 172.21.15.45:0/2635755240 learned_addr learned my addr 172.21.15.45:0/2635755240 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 10 client.?.objecter ms_handle_connect 0x7f64c413f650 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212641455 0 0) 0x7f64c413c860 con 0x7f64c413f650 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.772+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b0004eb0 con 0x7f64c413f650 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1621736054 0 0) 0x7f64b0004eb0 con 0x7f64c413f650 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c413c860 con 0x7f64c413f650 2024-03-29T10:55:59.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f64b4001950 con 0x7f64c413f650 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 10 client.?.objecter ms_handle_connect 0x7f64c41457f0 2024-03-29T10:55:59.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 439614293 0 0) 0x7f64c413c860 con 0x7f64c413f650 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.773+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 >> v1:172.21.15.45:6790/0 conn(0x7f64c40a9d70 legacy=0x7f64c4070e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 >> v1:172.21.15.45:6789/0 conn(0x7f64c41457f0 legacy=0x7f64c41e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 1 monclient: found mon.b 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c41396a0 con 0x7f64c413f650 2024-03-29T10:55:59.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 10 client.?.objecter ms_handle_connect 0x7f64c40a9d70 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64c921e640 5 monclient: authenticate success, global_id 4384 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64c921e640 10 monclient: _renew_subs 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64c921e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.775+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c4070340 con 0x7f64c413f650 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64c921e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64c921e640 10 monclient: _renew_subs 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64c921e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c40b0e90 con 0x7f64c413f650 2024-03-29T10:55:59.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64c921e640 1 librados: init done 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b40047e0 con 0x7f64c413f650 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f64b4001950 con 0x7f64c413f650 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64ab7fe640 10 monclient: dump: 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:55:59.758 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:55:59.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.776+0000 7f64c0d99640 4 set_mon_vals no callback set 2024-03-29T10:55:59.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.778+0000 7f64c921e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.778+0000 7f64c921e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.778+0000 7f64c921e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.778+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a4005080 con 0x7f64c413f650 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.778+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f64b4004950 con 0x7f64c413f650 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64c2d9d640 10 monclient: get_auth_request con 0x7f64b006dde0 auth_method 0 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f64b40546a0 con 0x7f64c413f650 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64ab7fe640 10 client.4384.objecter ms_dispatch 0x7f64c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64ab7fe640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64ab7fe640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64ab7fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T10:55:59.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.780+0000 7f64ab7fe640 10 client.4384.objecter ms_handle_connect 0x7f64b006dde0 2024-03-29T10:55:59.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.791+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64b4050940 con 0x7f64c413f650 2024-03-29T10:55:59.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.791+0000 7f64ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:59.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:55:59.791+0000 7f64ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:00.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.120+0000 7f64c921e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:00.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.120+0000 7f64c921e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:00.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.120+0000 7f64c921e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:00.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.120+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64a4005330 con 0x7f64c413f650 2024-03-29T10:56:00.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.121+0000 7f64ab7fe640 1 -- 172.21.15.45:0/2635755240 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f64b4050940 con 0x7f64c413f650 2024-03-29T10:56:00.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.121+0000 7f64ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:00.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.121+0000 7f64ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:00.102 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:00.102 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.124+0000 7f64c921e640 10 librados: watch_flush enter 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.124+0000 7f64c921e640 10 librados: watch_flush exit 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.124+0000 7f64c921e640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.124+0000 7f64c921e640 10 client.4384.objecter successfully canceled tick 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.125+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 >> v1:172.21.15.116:6816/3290579615 conn(0x7f64b006dde0 legacy=0x7f64b00702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.125+0000 7f64c921e640 10 monclient: shutdown 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.125+0000 7f64c921e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.125+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 >> v1:172.21.15.116:6789/0 conn(0x7f64c413f650 legacy=0x7f64c406f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.125+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 shutdown_connections 2024-03-29T10:56:00.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.125+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 >> 172.21.15.45:0/2635755240 conn(0x7f64c4066a00 msgr2=0x7f64c40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:00.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.127+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 shutdown_connections 2024-03-29T10:56:00.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.127+0000 7f64c921e640 1 -- 172.21.15.45:0/2635755240 wait complete. 2024-03-29T10:56:00.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:00.127+0000 7f64c921e640 1 librados: shutdown 2024-03-29T10:56:00.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:06.929 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:07.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.095+0000 7f65fe14d640 10 monclient: get_monmap_and_config 2024-03-29T10:56:07.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.095+0000 7f65fe14d640 10 monclient: build_initial_monmap 2024-03-29T10:56:07.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.095+0000 7f65fe14d640 10 monclient: monmap: 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:07.096668+0000 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:07.096668+0000 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.097+0000 7f65fe14d640 1 Processor -- start 2024-03-29T10:56:07.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.097+0000 7f65fe14d640 1 -- start start 2024-03-29T10:56:07.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.097+0000 7f65fe14d640 10 monclient: init 2024-03-29T10:56:07.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient(hunting): picked mon.noname-b con 0x7f65f002cc50 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient(hunting): picked mon.noname-c con 0x7f65f002d3a0 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient(hunting): picked mon.noname-a con 0x7f65f00c6290 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00c56f0 con 0x7f65f00c6290 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f002e080 con 0x7f65f002d3a0 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f002e360 con 0x7f65f002cc50 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.098+0000 7f65fe14d640 10 monclient(hunting): authenticate will time out at 1041.367730s 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65fc94a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65f002d3a0 0x7f65f00c5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:35608/0 (socket says 172.21.15.45:35608) 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65fc94a640 1 -- 172.21.15.45:0/2916014908 learned_addr learned my addr 172.21.15.45:0/2916014908 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555634005 0 0) 0x7f65f002e080 con 0x7f65f002d3a0 2024-03-29T10:56:07.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65effff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65effff640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.099+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e4003520 con 0x7f65f002d3a0 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.101+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2094040421 0 0) 0x7f65e4003520 con 0x7f65f002d3a0 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.101+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 >> v1:172.21.15.45:6790/0 conn(0x7f65f002cc50 legacy=0x7f65f00c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.101+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 >> v1:172.21.15.45:6789/0 conn(0x7f65f00c6290 legacy=0x7f65f00d0fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 1 monclient: found mon.noname-c 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00c8c10 con 0x7f65f002d3a0 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 10 monclient: _finish_auth 0 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:07.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65fe14d640 5 monclient: authenticate success, global_id 4387 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65fe14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f65e8001670 con 0x7f65f002d3a0 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.102+0000 7f65effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 10 monclient: dump: 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:07.086 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:07.087 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65fe14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:07.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65e8001900 con 0x7f65f002d3a0 2024-03-29T10:56:07.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:07.089 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65fe14d640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:07.089 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65fe14d640 4 set_mon_vals no callback set 2024-03-29T10:56:07.089 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 1 -- 172.21.15.45:0/2916014908 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f65e8001c90 con 0x7f65f002d3a0 2024-03-29T10:56:07.089 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.089 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.103+0000 7f65effff640 10 monclient: dump: 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.104+0000 7f65fe14d640 10 monclient: shutdown 2024-03-29T10:56:07.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.104+0000 7f65fe14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.104+0000 7f65fe14d640 1 -- 172.21.15.45:0/2916014908 >> v1:172.21.15.116:6789/0 conn(0x7f65f002d3a0 legacy=0x7f65f00c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.104+0000 7f65fe14d640 1 -- 172.21.15.45:0/2916014908 shutdown_connections 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.104+0000 7f65fe14d640 1 -- 172.21.15.45:0/2916014908 >> 172.21.15.45:0/2916014908 conn(0x7f65f00058f0 msgr2=0x7f65f0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.105+0000 7f65fe14d640 1 -- 172.21.15.45:0/2916014908 shutdown_connections 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.106+0000 7f65fe14d640 1 -- 172.21.15.45:0/2916014908 wait complete. 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 10 monclient: build_initial_monmap 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 10 monclient: monmap: 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.091 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 1 librados: starting msgr at 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 1 librados: starting objecter 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 1 Processor -- start 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 1 -- start start 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 1 librados: setting wanted keys 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 1 librados: calling monclient init 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.107+0000 7f65fe14d640 10 monclient: init 2024-03-29T10:56:07.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.108+0000 7f65fe14d640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:07.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.108+0000 7f65fe14d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:07.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.108+0000 7f65fe14d640 10 monclient(hunting): picked mon.noname-a con 0x7f65f002cc50 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:07.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 10 monclient(hunting): picked mon.noname-c con 0x7f65f002d3a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:07.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 10 monclient(hunting): picked mon.noname-b con 0x7f65f00c6290 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:07.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00d6760 con 0x7f65f002cc50 2024-03-29T10:56:07.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00d5570 con 0x7f65f00c6290 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00d6380 con 0x7f65f002d3a0 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.111+0000 7f65fe14d640 10 monclient(hunting): authenticate will time out at 1041.380797s 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.112+0000 7f65fd94c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65f00c6290 0x7f65f016b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:35888/0 (socket says 172.21.15.45:35888) 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.112+0000 7f65fd94c640 1 -- 172.21.15.45:0/1756736564 learned_addr learned my addr 172.21.15.45:0/1756736564 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 10 client.?.objecter ms_handle_connect 0x7f65f00c6290 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121592972 0 0) 0x7f65f00d5570 con 0x7f65f00c6290 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.113+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65dc004eb0 con 0x7f65f00c6290 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 client.?.objecter ms_handle_connect 0x7f65f002cc50 2024-03-29T10:56:07.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388963035 0 0) 0x7f65f00d6760 con 0x7f65f002cc50 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f00d5570 con 0x7f65f002cc50 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 client.?.objecter ms_handle_connect 0x7f65f002d3a0 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806641672 0 0) 0x7f65f00d6380 con 0x7f65f002d3a0 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.114+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f00d6760 con 0x7f65f002d3a0 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.115+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4019511656 0 0) 0x7f65f00d5570 con 0x7f65f002cc50 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.115+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f00d6380 con 0x7f65f002cc50 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.115+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908765285 0 0) 0x7f65f00d6760 con 0x7f65f002d3a0 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.115+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f00d5570 con 0x7f65f002d3a0 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.115+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f65f4001510 con 0x7f65f002cc50 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.115+0000 7f65edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 10 monclient(hunting): dump: 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.103 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f65e8001530 con 0x7f65f002d3a0 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 10 monclient(hunting): dump: 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.104 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2933171942 0 0) 0x7f65dc004eb0 con 0x7f65f00c6290 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f00d6760 con 0x7f65f00c6290 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.116+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 925635518 0 0) 0x7f65f00d6380 con 0x7f65f002cc50 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 >> v1:172.21.15.45:6790/0 conn(0x7f65f002d3a0 legacy=0x7f65f00d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 >> v1:172.21.15.116:6789/0 conn(0x7f65f00c6290 legacy=0x7f65f016b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 1 monclient: found mon.a 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00d3460 con 0x7f65f002cc50 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:07.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65fe14d640 5 monclient: authenticate success, global_id 4365 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65fe14d640 10 monclient: _renew_subs 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65fe14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.117+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f00d3710 con 0x7f65f002cc50 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.118+0000 7f65fe14d640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.118+0000 7f65fe14d640 10 monclient: _renew_subs 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.118+0000 7f65fe14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.118+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f00d4490 con 0x7f65f002cc50 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.118+0000 7f65fe14d640 1 librados: init done 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.119+0000 7f65fe14d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.119+0000 7f65fe14d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.119+0000 7f65fe14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:56:07.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00d6510 con 0x7f65f002cc50 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65f4001670 con 0x7f65f002cc50 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65ef7fe640 4 set_mon_vals no callback set 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f65f4001d00 con 0x7f65f002cc50 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.120+0000 7f65edffb640 10 monclient: dump: 2024-03-29T10:56:07.107 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.121+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f65f4019780 con 0x7f65f002cc50 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.122+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f65f40835c0 con 0x7f65f002cc50 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.122+0000 7f65edffb640 10 client.4365.objecter ms_dispatch 0x7f65f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.122+0000 7f65edffb640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.122+0000 7f65edffb640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.122+0000 7f65edffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.122+0000 7f65fc94a640 10 monclient: get_auth_request con 0x7f65dc06e030 auth_method 0 2024-03-29T10:56:07.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.124+0000 7f65edffb640 10 client.4365.objecter ms_handle_connect 0x7f65dc06e030 2024-03-29T10:56:07.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.130+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65f404ef00 con 0x7f65f002cc50 2024-03-29T10:56:07.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.130+0000 7f65edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:07.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.130+0000 7f65edffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.462+0000 7f65fe14d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.462+0000 7f65fe14d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.462+0000 7f65fe14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.462+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65f00d4730 con 0x7f65f002cc50 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.463+0000 7f65edffb640 1 -- 172.21.15.45:0/1756736564 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f65f4001b60 con 0x7f65f002cc50 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.463+0000 7f65edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:07.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.463+0000 7f65edffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:07.444 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:07.444 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:07.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 10 librados: watch_flush enter 2024-03-29T10:56:07.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 10 librados: watch_flush exit 2024-03-29T10:56:07.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-29T10:56:07.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 10 client.4365.objecter successfully canceled tick 2024-03-29T10:56:07.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 >> v1:172.21.15.116:6816/3290579615 conn(0x7f65dc06e030 legacy=0x7f65dc0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 10 monclient: shutdown 2024-03-29T10:56:07.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:07.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 >> v1:172.21.15.45:6789/0 conn(0x7f65f002cc50 legacy=0x7f65f00d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 shutdown_connections 2024-03-29T10:56:07.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.467+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 >> 172.21.15.45:0/1756736564 conn(0x7f65f00058f0 msgr2=0x7f65f00043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:07.451 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.470+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 shutdown_connections 2024-03-29T10:56:07.451 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.471+0000 7f65fe14d640 1 -- 172.21.15.45:0/1756736564 wait complete. 2024-03-29T10:56:07.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:07.471+0000 7f65fe14d640 1 librados: shutdown 2024-03-29T10:56:07.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:14.279 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.446+0000 7ffb1b143640 10 monclient: get_monmap_and_config 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.446+0000 7ffb1b143640 10 monclient: build_initial_monmap 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.446+0000 7ffb1b143640 10 monclient: monmap: 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:14.447223+0000 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:14.447223+0000 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:14.427 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.447+0000 7ffb1b143640 1 Processor -- start 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.447+0000 7ffb1b143640 1 -- start start 2024-03-29T10:56:14.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.447+0000 7ffb1b143640 10 monclient: init 2024-03-29T10:56:14.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:14.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:14.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient(hunting): picked mon.noname-c con 0x7ffb140b1f60 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient(hunting): picked mon.noname-b con 0x7ffb14071eb0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient(hunting): picked mon.noname-a con 0x7ffb14143d10 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1406ef80 con 0x7ffb14143d10 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1406f260 con 0x7ffb140b1f60 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140b40c0 con 0x7ffb14071eb0 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:14.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.448+0000 7ffb1b143640 10 monclient(hunting): authenticate will time out at 1048.717610s 2024-03-29T10:56:14.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.451+0000 7ffb18edf640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffb140b1f60 0x7ffb14143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:36118/0 (socket says 172.21.15.45:36118) 2024-03-29T10:56:14.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.451+0000 7ffb18edf640 1 -- 172.21.15.45:0/1347260033 learned_addr learned my addr 172.21.15.45:0/1347260033 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:14.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.452+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070424567 0 0) 0x7ffb1406f260 con 0x7ffb140b1f60 2024-03-29T10:56:14.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.452+0000 7ffb137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.452+0000 7ffb137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.452+0000 7ffb137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.452+0000 7ffb137fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.452+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc003520 con 0x7ffb140b1f60 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.453+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421124163 0 0) 0x7ffafc003520 con 0x7ffb140b1f60 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.453+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 >> v1:172.21.15.45:6790/0 conn(0x7ffb14071eb0 legacy=0x7ffb14146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.453+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 >> v1:172.21.15.45:6789/0 conn(0x7ffb14143d10 legacy=0x7ffb14146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 1 monclient: found mon.noname-c 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb14155d70 con 0x7ffb140b1f60 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:14.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ffb04001690 con 0x7ffb140b1f60 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: dump: 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:14.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb04001860 con 0x7ffb140b1f60 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 1 -- 172.21.15.45:0/1347260033 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ffb04001cb0 con 0x7ffb140b1f60 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.454+0000 7ffb137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.455+0000 7ffb137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.455+0000 7ffb137fe640 10 monclient: dump: 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:14.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.456+0000 7ffb1b143640 5 monclient: authenticate success, global_id 4393 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.456+0000 7ffb1b143640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.456+0000 7ffb1b143640 4 set_mon_vals no callback set 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.456+0000 7ffb1b143640 10 monclient: shutdown 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.456+0000 7ffb1b143640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.456+0000 7ffb1b143640 1 -- 172.21.15.45:0/1347260033 >> v1:172.21.15.116:6789/0 conn(0x7ffb140b1f60 legacy=0x7ffb14143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.457+0000 7ffb1b143640 1 -- 172.21.15.45:0/1347260033 shutdown_connections 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.457+0000 7ffb1b143640 1 -- 172.21.15.45:0/1347260033 >> 172.21.15.45:0/1347260033 conn(0x7ffb14066a00 msgr2=0x7ffb14066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.457+0000 7ffb1b143640 1 -- 172.21.15.45:0/1347260033 shutdown_connections 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.457+0000 7ffb1b143640 1 -- 172.21.15.45:0/1347260033 wait complete. 2024-03-29T10:56:14.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 10 monclient: build_initial_monmap 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 10 monclient: monmap: 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 1 librados: starting msgr at 2024-03-29T10:56:14.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 1 librados: starting objecter 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 1 Processor -- start 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 1 -- start start 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 1 librados: setting wanted keys 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 1 librados: calling monclient init 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.458+0000 7ffb1b143640 10 monclient: init 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient(hunting): picked mon.noname-c con 0x7ffb140b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient(hunting): picked mon.noname-b con 0x7ffb14143d10 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient(hunting): picked mon.noname-a con 0x7ffb141e4420 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb14140720 con 0x7ffb141e4420 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140ace60 con 0x7ffb14143d10 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140ad140 con 0x7ffb140b1f60 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:14.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.459+0000 7ffb1b143640 10 monclient(hunting): authenticate will time out at 1048.728863s 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.463+0000 7ffb18edf640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ffb140b1f60 0x7ffb141e3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58755/0 (socket says 172.21.15.45:58755) 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.463+0000 7ffb18edf640 1 -- 172.21.15.45:0/556061317 learned_addr learned my addr 172.21.15.45:0/556061317 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 client.?.objecter ms_handle_connect 0x7ffb140b1f60 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 client.?.objecter ms_handle_connect 0x7ffb14143d10 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644100622 0 0) 0x7ffb140ad140 con 0x7ffb140b1f60 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:14.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.464+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb00004eb0 con 0x7ffb140b1f60 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359388035 0 0) 0x7ffb140ace60 con 0x7ffb14143d10 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb140ad140 con 0x7ffb14143d10 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684643581 0 0) 0x7ffb00004eb0 con 0x7ffb140b1f60 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb000054e0 con 0x7ffb140b1f60 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ffb04001b70 con 0x7ffb140b1f60 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): dump: 2024-03-29T10:56:14.447 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 537504646 0 0) 0x7ffb000054e0 con 0x7ffb140b1f60 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 >> v1:172.21.15.116:6789/0 conn(0x7ffb14143d10 legacy=0x7ffb1413f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 >> v1:172.21.15.45:6789/0 conn(0x7ffb141e4420 legacy=0x7ffb14140010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:14.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 monclient: found mon.c 2024-03-29T10:56:14.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:14.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:14.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140ab9a0 con 0x7ffb140b1f60 2024-03-29T10:56:14.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:14.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:14.449 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.465+0000 7ffb117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb04004bf0 con 0x7ffb140b1f60 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ffb04001b70 con 0x7ffb140b1f60 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb117fa640 10 monclient: dump: 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:14.452 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 10 monclient: _renew_subs 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb14140a10 con 0x7ffb140b1f60 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 10 monclient: _renew_subs 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb140ab760 con 0x7ffb140b1f60 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.466+0000 7ffb1b143640 1 librados: init done 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.467+0000 7ffb1b143640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.467+0000 7ffb1b143640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.467+0000 7ffb1b143640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.467+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb14075f10 con 0x7ffb140b1f60 2024-03-29T10:56:14.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.468+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7ffb0401abd0 con 0x7ffb140b1f60 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.469+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7ffb04084750 con 0x7ffb140b1f60 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.469+0000 7ffb117fa640 10 client.4379.objecter ms_dispatch 0x7ffb14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.469+0000 7ffb117fa640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.469+0000 7ffb117fa640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.469+0000 7ffb13fff640 10 monclient: get_auth_request con 0x7ffb0006e180 auth_method 0 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.469+0000 7ffb117fa640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.468+0000 7ffb12ffd640 4 set_mon_vals no callback set 2024-03-29T10:56:14.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.470+0000 7ffb117fa640 10 client.4379.objecter ms_handle_connect 0x7ffb0006e180 2024-03-29T10:56:14.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.476+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb04050030 con 0x7ffb140b1f60 2024-03-29T10:56:14.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.476+0000 7ffb117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:14.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.476+0000 7ffb117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.792+0000 7ffb1b143640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.792+0000 7ffb1b143640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.792+0000 7ffb1b143640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.792+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb140ab760 con 0x7ffb140b1f60 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.793+0000 7ffb117fa640 1 -- 172.21.15.45:0/556061317 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7ffb04051f60 con 0x7ffb140b1f60 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.793+0000 7ffb117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.793+0000 7ffb117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 10 librados: watch_flush enter 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 10 librados: watch_flush exit 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 >> v1:172.21.15.116:6816/3290579615 conn(0x7ffb0006e180 legacy=0x7ffb00070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 10 monclient: shutdown 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 >> v1:172.21.15.45:6790/0 conn(0x7ffb140b1f60 legacy=0x7ffb141e3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 shutdown_connections 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 >> 172.21.15.45:0/556061317 conn(0x7ffb14066a00 msgr2=0x7ffb14142a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 shutdown_connections 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 -- 172.21.15.45:0/556061317 wait complete. 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:14.796+0000 7ffb1b143640 1 librados: shutdown 2024-03-29T10:56:14.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:21.598 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.764+0000 7f8fef59e640 10 monclient: get_monmap_and_config 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.764+0000 7f8fef59e640 10 monclient: build_initial_monmap 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.764+0000 7f8fef59e640 10 monclient: monmap: 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:21.766420+0000 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:21.766420+0000 2024-03-29T10:56:21.746 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.765+0000 7f8fef59e640 1 Processor -- start 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.766+0000 7f8fef59e640 1 -- start start 2024-03-29T10:56:21.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.766+0000 7f8fef59e640 10 monclient: init 2024-03-29T10:56:21.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:21.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe8006eb0 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe8038f10 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe8036690 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe8033c90 con 0x7f8fe8038f10 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80389f0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe8030890 con 0x7f8fe8036690 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fef59e640 10 monclient(hunting): authenticate will time out at 1056.037444s 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fedd9b640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8fe8038f10 0x7f8fe8039330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49052/0 (socket says 172.21.15.45:49052) 2024-03-29T10:56:21.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.767+0000 7f8fedd9b640 1 -- 172.21.15.45:0/2463563125 learned_addr learned my addr 172.21.15.45:0/2463563125 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:21.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.768+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318578107 0 0) 0x7f8fe80389f0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.768+0000 7f8fed59a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.768+0000 7f8fed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.768+0000 7f8fed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.768+0000 7f8fed59a640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:56:21.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.768+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd8003520 con 0x7f8fe8006eb0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182614594 0 0) 0x7f8fd8003520 con 0x7f8fe8006eb0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 >> v1:172.21.15.45:6790/0 conn(0x7f8fe8036690 legacy=0x7f8fe8036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 >> v1:172.21.15.45:6789/0 conn(0x7f8fe8038f10 legacy=0x7f8fe8039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 1 monclient: found mon.noname-c 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80cbc40 con 0x7f8fe8006eb0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: _finish_auth 0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f8fe00015c0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.751 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fed59a640 10 monclient: dump: 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fef59e640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:56:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.769+0000 7f8fef59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:21.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fe0001790 con 0x7f8fe8006eb0 2024-03-29T10:56:21.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:21.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fed59a640 1 -- 172.21.15.45:0/2463563125 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f8fe0001ba0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fef59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fef59e640 4 set_mon_vals no callback set 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.770+0000 7f8fed59a640 10 monclient: dump: 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:21.754 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.771+0000 7f8fef59e640 10 monclient: shutdown 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.771+0000 7f8fef59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.771+0000 7f8fef59e640 1 -- 172.21.15.45:0/2463563125 >> v1:172.21.15.116:6789/0 conn(0x7f8fe8006eb0 legacy=0x7f8fe8031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.771+0000 7f8fef59e640 1 -- 172.21.15.45:0/2463563125 shutdown_connections 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.771+0000 7f8fef59e640 1 -- 172.21.15.45:0/2463563125 >> 172.21.15.45:0/2463563125 conn(0x7f8fe8005c20 msgr2=0x7f8fe8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.771+0000 7f8fef59e640 1 -- 172.21.15.45:0/2463563125 shutdown_connections 2024-03-29T10:56:21.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.772+0000 7f8fef59e640 1 -- 172.21.15.45:0/2463563125 wait complete. 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 10 monclient: build_initial_monmap 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 10 monclient: monmap: 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 1 librados: starting msgr at 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 1 librados: starting objecter 2024-03-29T10:56:21.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 1 Processor -- start 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 1 -- start start 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 1 librados: setting wanted keys 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 1 librados: calling monclient init 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.773+0000 7f8fef59e640 10 monclient: init 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.774+0000 7f8fef59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.774+0000 7f8fef59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.774+0000 7f8fef59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe8006eb0 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.774+0000 7f8fef59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe8036690 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fef59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe8038f10 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fee59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8fe8006eb0 0x7f8fe80cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:47846/0 (socket says 172.21.15.45:47846) 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fee59c640 1 -- 172.21.15.45:0/1658855367 learned_addr learned my addr 172.21.15.45:0/1658855367 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe815d8d0 con 0x7f8fe8036690 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80c8f00 con 0x7f8fe8006eb0 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80d0000 con 0x7f8fe8038f10 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fef59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fef59e640 10 monclient(hunting): authenticate will time out at 1056.045142s 2024-03-29T10:56:21.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fdeffd640 10 client.?.objecter ms_handle_connect 0x7f8fe8006eb0 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fdeffd640 10 client.?.objecter ms_handle_connect 0x7f8fe8038f10 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.775+0000 7f8fdeffd640 10 client.?.objecter ms_handle_connect 0x7f8fe8036690 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314962842 0 0) 0x7f8fe80c8f00 con 0x7f8fe8006eb0 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd0004eb0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286481246 0 0) 0x7f8fe815d8d0 con 0x7f8fe8036690 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:56:21.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe80c8f00 con 0x7f8fe8036690 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570695812 0 0) 0x7f8fd0004eb0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe815d8d0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f8fe0001b60 con 0x7f8fe8006eb0 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): dump: 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:21.766 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432487981 0 0) 0x7f8fe80d0000 con 0x7f8fe8038f10 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.776+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd0004eb0 con 0x7f8fe8038f10 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 242497545 0 0) 0x7f8fe815d8d0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 >> v1:172.21.15.45:6790/0 conn(0x7f8fe8038f10 legacy=0x7f8fe8160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 >> v1:172.21.15.45:6789/0 conn(0x7f8fe8036690 legacy=0x7f8fe80c09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 1 monclient: found mon.b 2024-03-29T10:56:21.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80c8b90 con 0x7f8fe8006eb0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 10 monclient: _renew_subs 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe802fe80 con 0x7f8fe8006eb0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 10 monclient: _renew_subs 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe80c1290 con 0x7f8fe8006eb0 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.777+0000 7f8fef59e640 1 librados: init done 2024-03-29T10:56:21.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.778+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fe00038a0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.778+0000 7f8fdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.778+0000 7f8fecd99640 4 set_mon_vals no callback set 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f8fe0003cb0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fdeffd640 10 monclient: dump: 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:21.769 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fef59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fef59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fef59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.779+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4005080 con 0x7f8fe8006eb0 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.780+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f8fe0003ed0 con 0x7f8fe8006eb0 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fedd9b640 10 monclient: get_auth_request con 0x7f8fd006dda0 auth_method 0 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f8fe0084b50 con 0x7f8fe8006eb0 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fdeffd640 10 client.4402.objecter ms_dispatch 0x7f8ff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fdeffd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fdeffd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fdeffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:56:21.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.781+0000 7f8fdeffd640 10 client.4402.objecter ms_handle_connect 0x7f8fd006dda0 2024-03-29T10:56:21.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.786+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fe0050430 con 0x7f8fe8006eb0 2024-03-29T10:56:21.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.787+0000 7f8fdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:21.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:21.787+0000 7f8fdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.143+0000 7f8fef59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.143+0000 7f8fef59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.143+0000 7f8fef59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.143+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd4005330 con 0x7f8fe8006eb0 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.144+0000 7f8fdeffd640 1 -- 172.21.15.45:0/1658855367 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f8fe0052360 con 0x7f8fe8006eb0 2024-03-29T10:56:22.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.144+0000 7f8fdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:22.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.144+0000 7f8fdeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 10 librados: watch_flush enter 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 10 librados: watch_flush exit 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 >> v1:172.21.15.116:6816/3290579615 conn(0x7f8fd006dda0 legacy=0x7f8fd0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 10 monclient: shutdown 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.148+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 >> v1:172.21.15.116:6789/0 conn(0x7f8fe8006eb0 legacy=0x7f8fe80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.149+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 shutdown_connections 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.149+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 >> 172.21.15.45:0/1658855367 conn(0x7f8fe8005c20 msgr2=0x7f8fe803c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.149+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 shutdown_connections 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.149+0000 7f8fef59e640 1 -- 172.21.15.45:0/1658855367 wait complete. 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:22.149+0000 7f8fef59e640 1 librados: shutdown 2024-03-29T10:56:22.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:28.955 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:29.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.128+0000 7f7ec816b640 10 monclient: get_monmap_and_config 2024-03-29T10:56:29.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.128+0000 7f7ec816b640 10 monclient: build_initial_monmap 2024-03-29T10:56:29.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.128+0000 7f7ec816b640 10 monclient: monmap: 2024-03-29T10:56:29.110 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:29.130709+0000 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:29.130709+0000 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:29.111 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:29.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.129+0000 7f7ec816b640 1 Processor -- start 2024-03-29T10:56:29.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.130+0000 7f7ec816b640 1 -- start start 2024-03-29T10:56:29.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.130+0000 7f7ec816b640 10 monclient: init 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec00708a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec006fe90 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec0143c50 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec0072630 con 0x7f7ec0143c50 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec0071410 con 0x7f7ec006fe90 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00afaa0 con 0x7f7ec00708a0 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec816b640 10 monclient(hunting): authenticate will time out at 1063.402283s 2024-03-29T10:56:29.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec5706640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7ec006fe90 0x7f7ec0143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:47566/0 (socket says 172.21.15.45:47566) 2024-03-29T10:56:29.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.132+0000 7f7ec5706640 1 -- 172.21.15.45:0/3142338178 learned_addr learned my addr 172.21.15.45:0/3142338178 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:29.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469955469 0 0) 0x7f7ec00afaa0 con 0x7f7ec00708a0 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eac003520 con 0x7f7ec00708a0 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726629355 0 0) 0x7f7ec0071410 con 0x7f7ec006fe90 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.133+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec00afaa0 con 0x7f7ec006fe90 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562835573 0 0) 0x7f7ec0072630 con 0x7f7ec0143c50 2024-03-29T10:56:29.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec0071410 con 0x7f7ec0143c50 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1271449166 0 0) 0x7f7ec00afaa0 con 0x7f7ec006fe90 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 >> v1:172.21.15.45:6790/0 conn(0x7f7ec00708a0 legacy=0x7f7ec00af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 >> v1:172.21.15.45:6789/0 conn(0x7f7ec0143c50 legacy=0x7f7ec014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: _finish_hunting 0 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 monclient: found mon.noname-c 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec00ac8b0 con 0x7f7ec006fe90 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: _finish_auth 0 2024-03-29T10:56:29.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: _check_auth_tickets 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec816b640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec816b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f7eb00015f0 con 0x7f7ec006fe90 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec4f05640 10 monclient: dump: 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:29.121 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.134+0000 7f7ec816b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb0001750 con 0x7f7ec006fe90 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec4f05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec4f05640 1 -- 172.21.15.45:0/3142338178 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f7eb0001b60 con 0x7f7ec006fe90 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec4f05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec4f05640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec4f05640 10 monclient: dump: 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:29.122 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec816b640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.135+0000 7f7ec816b640 4 set_mon_vals no callback set 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.136+0000 7f7ec816b640 10 monclient: shutdown 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.136+0000 7f7ec816b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.136+0000 7f7ec816b640 1 -- 172.21.15.45:0/3142338178 >> v1:172.21.15.116:6789/0 conn(0x7f7ec006fe90 legacy=0x7f7ec0143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.136+0000 7f7ec816b640 1 -- 172.21.15.45:0/3142338178 shutdown_connections 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.136+0000 7f7ec816b640 1 -- 172.21.15.45:0/3142338178 >> 172.21.15.45:0/3142338178 conn(0x7f7ec00675b0 msgr2=0x7f7ec00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.137+0000 7f7ec816b640 1 -- 172.21.15.45:0/3142338178 shutdown_connections 2024-03-29T10:56:29.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.137+0000 7f7ec816b640 1 -- 172.21.15.45:0/3142338178 wait complete. 2024-03-29T10:56:29.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.138+0000 7f7ec816b640 10 monclient: build_initial_monmap 2024-03-29T10:56:29.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.138+0000 7f7ec816b640 10 monclient: monmap: 2024-03-29T10:56:29.129 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:29.129 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:29.129 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.138+0000 7f7ec816b640 1 librados: starting msgr at 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.138+0000 7f7ec816b640 1 librados: starting objecter 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.139+0000 7f7ec816b640 1 Processor -- start 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.139+0000 7f7ec816b640 1 -- start start 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.139+0000 7f7ec816b640 1 librados: setting wanted keys 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.139+0000 7f7ec816b640 1 librados: calling monclient init 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.139+0000 7f7ec816b640 10 monclient: init 2024-03-29T10:56:29.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.139+0000 7f7ec816b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec006fe90 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec00708a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec0143c50 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00aae70 con 0x7f7ec006fe90 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00abd90 con 0x7f7ec0143c50 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00aab90 con 0x7f7ec00708a0 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec816b640 10 monclient(hunting): authenticate will time out at 1063.410258s 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec5f07640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f7ec006fe90 0x7f7ec00aa480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38604/0 (socket says 172.21.15.45:38604) 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7ec5f07640 1 -- 172.21.15.45:0/213382221 learned_addr learned my addr 172.21.15.45:0/213382221 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7eb67fc640 10 client.?.objecter ms_handle_connect 0x7f7ec006fe90 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7eb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7eb67fc640 10 client.?.objecter ms_handle_connect 0x7f7ec0143c50 2024-03-29T10:56:29.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.140+0000 7f7eb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968498322 0 0) 0x7f7ec00abd90 con 0x7f7ec0143c50 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea4004eb0 con 0x7f7ec0143c50 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 client.?.objecter ms_handle_connect 0x7f7ec00708a0 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187971231 0 0) 0x7f7ec00aab90 con 0x7f7ec00708a0 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec00abd90 con 0x7f7ec00708a0 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3402185341 0 0) 0x7f7ea4004eb0 con 0x7f7ec0143c50 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ea4005490 con 0x7f7ec0143c50 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f7eb8001bf0 con 0x7f7ec0143c50 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): dump: 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:29.133 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.141+0000 7f7eb67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964134295 0 0) 0x7f7ec00aae70 con 0x7f7ec006fe90 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea4004eb0 con 0x7f7ec006fe90 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2187436842 0 0) 0x7f7ea4005490 con 0x7f7ec0143c50 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 >> v1:172.21.15.45:6790/0 conn(0x7f7ec00708a0 legacy=0x7f7ec013c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 >> v1:172.21.15.45:6789/0 conn(0x7f7ec006fe90 legacy=0x7f7ec00aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 monclient: found mon.b 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec00a9a10 con 0x7f7ec0143c50 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7eb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 5 monclient: authenticate success, global_id 4408 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 10 monclient: _renew_subs 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec0140cd0 con 0x7f7ec0143c50 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 10 monclient: _renew_subs 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:29.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec013cc20 con 0x7f7ec0143c50 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.142+0000 7f7ec816b640 1 librados: init done 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.144+0000 7f7ec816b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.144+0000 7f7ec816b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.144+0000 7f7ec816b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.144+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec0071e40 con 0x7f7ec0143c50 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb80020e0 con 0x7f7ec0143c50 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f7eb8003b30 con 0x7f7ec0143c50 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb67fc640 10 monclient: dump: 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:29.136 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.145+0000 7f7eb7fff640 4 set_mon_vals no callback set 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.146+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f7eb8003d10 con 0x7f7ec0143c50 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.146+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f7eb8083b30 con 0x7f7ec0143c50 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.146+0000 7f7eb67fc640 10 client.4408.objecter ms_dispatch 0x7f7ec0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.146+0000 7f7eb67fc640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.146+0000 7f7eb67fc640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.146+0000 7f7eb67fc640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.147+0000 7f7ec5f07640 10 monclient: get_auth_request con 0x7f7ea406e040 auth_method 0 2024-03-29T10:56:29.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.147+0000 7f7eb67fc640 10 client.4408.objecter ms_handle_connect 0x7f7ea406e040 2024-03-29T10:56:29.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.152+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7eb804f240 con 0x7f7ec0143c50 2024-03-29T10:56:29.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.152+0000 7f7eb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:29.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.152+0000 7f7eb67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:29.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.548+0000 7f7ec816b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:29.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.548+0000 7f7ec816b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:29.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.548+0000 7f7ec816b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:56:29.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.548+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec0075f10 con 0x7f7ec0143c50 2024-03-29T10:56:29.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.549+0000 7f7eb67fc640 1 -- 172.21.15.45:0/213382221 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f7eb8051340 con 0x7f7ec0143c50 2024-03-29T10:56:29.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.549+0000 7f7eb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:29.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.549+0000 7f7eb67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:29.531 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:29.531 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:29.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.553+0000 7f7ec816b640 10 librados: watch_flush enter 2024-03-29T10:56:29.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.553+0000 7f7ec816b640 10 librados: watch_flush exit 2024-03-29T10:56:29.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.553+0000 7f7ec816b640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T10:56:29.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.553+0000 7f7ec816b640 10 client.4408.objecter successfully canceled tick 2024-03-29T10:56:29.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.553+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 >> v1:172.21.15.116:6816/3290579615 conn(0x7f7ea406e040 legacy=0x7f7ea4070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 10 monclient: shutdown 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 >> v1:172.21.15.116:6789/0 conn(0x7f7ec0143c50 legacy=0x7f7ec01deee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 shutdown_connections 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 >> 172.21.15.45:0/213382221 conn(0x7f7ec00675b0 msgr2=0x7f7ec00b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 shutdown_connections 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 1 -- 172.21.15.45:0/213382221 wait complete. 2024-03-29T10:56:29.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:29.554+0000 7f7ec816b640 1 librados: shutdown 2024-03-29T10:56:29.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:36.360 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.517+0000 7f2e6247b640 10 monclient: get_monmap_and_config 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.517+0000 7f2e6247b640 10 monclient: build_initial_monmap 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.517+0000 7f2e6247b640 10 monclient: monmap: 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:36.517998+0000 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:36.517998+0000 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.498 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:36.499 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:36.499 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:36.499 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.518+0000 7f2e6247b640 1 Processor -- start 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.518+0000 7f2e6247b640 1 -- start start 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.518+0000 7f2e6247b640 10 monclient: init 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.518+0000 7f2e6247b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.518+0000 7f2e6247b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.518+0000 7f2e6247b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c0b0860 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c072450 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c13ee40 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0ad840 con 0x7f2e5c072450 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c13c830 con 0x7f2e5c13ee40 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c13c9c0 con 0x7f2e5c0b0860 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e6247b640 10 monclient(hunting): authenticate will time out at 1070.788305s 2024-03-29T10:56:36.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e5bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2e5c0b0860 0x7f2e5c0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57034/0 (socket says 172.21.15.45:57034) 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e5bfff640 1 -- 172.21.15.45:0/4200598470 learned_addr learned my addr 172.21.15.45:0/4200598470 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.519+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394322379 0 0) 0x7f2e5c13c9c0 con 0x7f2e5c0b0860 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e44003520 con 0x7f2e5c0b0860 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204022048 0 0) 0x7f2e5c0ad840 con 0x7f2e5c072450 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:56:36.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c13c9c0 con 0x7f2e5c072450 2024-03-29T10:56:36.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130844203 0 0) 0x7f2e5c13c9c0 con 0x7f2e5c072450 2024-03-29T10:56:36.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 >> v1:172.21.15.45:6790/0 conn(0x7f2e5c0b0860 legacy=0x7f2e5c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.520+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 >> v1:172.21.15.116:6789/0 conn(0x7f2e5c13ee40 legacy=0x7f2e5c13f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 1 monclient: found mon.noname-a 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c13f970 con 0x7f2e5c072450 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 10 monclient: _finish_auth 0 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:36.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e6247b640 5 monclient: authenticate success, global_id 4380 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e6247b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.522+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2e50001670 con 0x7f2e5c072450 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e5affd640 10 monclient: dump: 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.507 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e6247b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e50001890 con 0x7f2e5c072450 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e5affd640 1 -- 172.21.15.45:0/4200598470 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2e50001c20 con 0x7f2e5c072450 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e6247b640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.523+0000 7f2e6247b640 4 set_mon_vals no callback set 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.524+0000 7f2e6247b640 10 monclient: shutdown 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.524+0000 7f2e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.524+0000 7f2e5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.524+0000 7f2e5affd640 10 monclient: dump: 2024-03-29T10:56:36.508 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.524+0000 7f2e6247b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.524+0000 7f2e6247b640 1 -- 172.21.15.45:0/4200598470 >> v1:172.21.15.45:6789/0 conn(0x7f2e5c072450 legacy=0x7f2e5c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.525+0000 7f2e6247b640 1 -- 172.21.15.45:0/4200598470 shutdown_connections 2024-03-29T10:56:36.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.525+0000 7f2e6247b640 1 -- 172.21.15.45:0/4200598470 >> 172.21.15.45:0/4200598470 conn(0x7f2e5c06b660 msgr2=0x7f2e5c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.525+0000 7f2e6247b640 1 -- 172.21.15.45:0/4200598470 shutdown_connections 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.525+0000 7f2e6247b640 1 -- 172.21.15.45:0/4200598470 wait complete. 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.527+0000 7f2e6247b640 10 monclient: build_initial_monmap 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.527+0000 7f2e6247b640 10 monclient: monmap: 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:36.511 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.527+0000 7f2e6247b640 1 librados: starting msgr at 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.527+0000 7f2e6247b640 1 librados: starting objecter 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.528+0000 7f2e6247b640 1 Processor -- start 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.528+0000 7f2e6247b640 1 -- start start 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.528+0000 7f2e6247b640 1 librados: setting wanted keys 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.528+0000 7f2e6247b640 1 librados: calling monclient init 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.528+0000 7f2e6247b640 10 monclient: init 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.529+0000 7f2e6247b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.529+0000 7f2e6247b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.529+0000 7f2e6247b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c072450 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c0b0860 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c13ee40 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:36.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c06e230 con 0x7f2e5c072450 2024-03-29T10:56:36.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c06f620 con 0x7f2e5c0b0860 2024-03-29T10:56:36.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c06f900 con 0x7f2e5c13ee40 2024-03-29T10:56:36.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:36.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e6247b640 10 monclient(hunting): authenticate will time out at 1070.799533s 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e5bfff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2e5c072450 0x7f2e5c0711d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57482/0 (socket says 172.21.15.45:57482) 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.530+0000 7f2e5bfff640 1 -- 172.21.15.45:0/101095717 learned_addr learned my addr 172.21.15.45:0/101095717 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 client.?.objecter ms_handle_connect 0x7f2e5c072450 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302368420 0 0) 0x7f2e5c06e230 con 0x7f2e5c072450 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:36.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48004eb0 con 0x7f2e5c072450 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 client.?.objecter ms_handle_connect 0x7f2e5c13ee40 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371393499 0 0) 0x7f2e5c06f900 con 0x7f2e5c13ee40 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c06e230 con 0x7f2e5c13ee40 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2511843257 0 0) 0x7f2e48004eb0 con 0x7f2e5c072450 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e5c06f900 con 0x7f2e5c072450 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669306131 0 0) 0x7f2e5c06e230 con 0x7f2e5c13ee40 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.531+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e48004eb0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2e4c001950 con 0x7f2e5c072450 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2e54001a10 con 0x7f2e5c13ee40 2024-03-29T10:56:36.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 208217718 0 0) 0x7f2e48004eb0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 >> v1:172.21.15.116:6789/0 conn(0x7f2e5c0b0860 legacy=0x7f2e5c0b11d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:36.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 >> v1:172.21.15.45:6789/0 conn(0x7f2e5c072450 legacy=0x7f2e5c0711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 1 monclient: found mon.c 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.532+0000 7f2e58ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c06e790 con 0x7f2e5c13ee40 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e54001b70 con 0x7f2e5c13ee40 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 10 monclient: _renew_subs 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c0b3b10 con 0x7f2e5c13ee40 2024-03-29T10:56:36.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e59ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2e54003970 con 0x7f2e5c13ee40 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e58ff9640 10 monclient: dump: 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:36.523 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 10 monclient: _renew_subs 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c071ac0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.533+0000 7f2e6247b640 1 librados: init done 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.535+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f2e540199c0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.536+0000 7f2e6247b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.536+0000 7f2e6247b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.536+0000 7f2e6247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.536+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c13a410 con 0x7f2e5c13ee40 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.536+0000 7f2e5bfff640 10 monclient: get_auth_request con 0x7f2e4806dde0 auth_method 0 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.537+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f2e540839c0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.537+0000 7f2e58ff9640 10 client.4397.objecter ms_dispatch 0x7f2e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.537+0000 7f2e58ff9640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.537+0000 7f2e58ff9640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.537+0000 7f2e58ff9640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.537+0000 7f2e58ff9640 10 client.4397.objecter ms_handle_connect 0x7f2e4806dde0 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.543+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e5404f2a0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.543+0000 7f2e58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:36.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.543+0000 7f2e58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.877+0000 7f2e6247b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.877+0000 7f2e6247b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.877+0000 7f2e6247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.877+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e5c0b0c40 con 0x7f2e5c13ee40 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.879+0000 7f2e58ff9640 1 -- 172.21.15.45:0/101095717 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f2e540511d0 con 0x7f2e5c13ee40 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.879+0000 7f2e58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.879+0000 7f2e58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:36.860 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 10 librados: watch_flush enter 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 10 librados: watch_flush exit 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 >> v1:172.21.15.116:6816/3290579615 conn(0x7f2e4806dde0 legacy=0x7f2e480702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 10 monclient: shutdown 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 >> v1:172.21.15.45:6790/0 conn(0x7f2e5c13ee40 legacy=0x7f2e5c1e3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 shutdown_connections 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 >> 172.21.15.45:0/101095717 conn(0x7f2e5c06b660 msgr2=0x7f2e5c13bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:36.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 shutdown_connections 2024-03-29T10:56:36.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 -- 172.21.15.45:0/101095717 wait complete. 2024-03-29T10:56:36.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:36.882+0000 7f2e6247b640 1 librados: shutdown 2024-03-29T10:56:36.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:43.686 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:43.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.864+0000 7f0d94d86640 10 monclient: get_monmap_and_config 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.864+0000 7f0d94d86640 10 monclient: build_initial_monmap 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.864+0000 7f0d94d86640 10 monclient: monmap: 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:43.865421+0000 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:43.865421+0000 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:43.846 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:43.847 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:43.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.868+0000 7f0d94d86640 1 Processor -- start 2024-03-29T10:56:43.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.868+0000 7f0d94d86640 1 -- start start 2024-03-29T10:56:43.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.868+0000 7f0d94d86640 10 monclient: init 2024-03-29T10:56:43.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8802b4c0 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8800b8a0 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8802a5a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8802c5c0 con 0x7f0d8802b4c0 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880c11a0 con 0x7f0d8800b8a0 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8802c830 con 0x7f0d8802a5a0 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:43.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.869+0000 7f0d94d86640 10 monclient(hunting): authenticate will time out at 1078.138264s 2024-03-29T10:56:43.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0d8802a5a0 0x7f0d880c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43211/0 (socket says 172.21.15.45:43211) 2024-03-29T10:56:43.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8ffff640 1 -- 172.21.15.45:0/3416917303 learned_addr learned my addr 172.21.15.45:0/3416917303 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:43.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133470320 0 0) 0x7f0d8802c830 con 0x7f0d8802a5a0 2024-03-29T10:56:43.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8e7fc640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:56:43.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.873+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d80003520 con 0x7f0d8802a5a0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 853213956 0 0) 0x7f0d80003520 con 0x7f0d8802a5a0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 >> v1:172.21.15.116:6789/0 conn(0x7f0d8800b8a0 legacy=0x7f0d880c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 >> v1:172.21.15.45:6789/0 conn(0x7f0d8802b4c0 legacy=0x7f0d88006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8802afa0 con 0x7f0d8802a5a0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0d9004f730 con 0x7f0d8802a5a0 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: dump: 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d900008d0 con 0x7f0d8802a5a0 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:43.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 1 -- 172.21.15.45:0/3416917303 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0d900758b0 con 0x7f0d8802a5a0 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d8e7fc640 10 monclient: dump: 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:43.857 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:43.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d94d86640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:56:43.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d94d86640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:43.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.874+0000 7f0d94d86640 4 set_mon_vals no callback set 2024-03-29T10:56:43.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.875+0000 7f0d94d86640 10 monclient: shutdown 2024-03-29T10:56:43.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.875+0000 7f0d94d86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:43.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.875+0000 7f0d94d86640 1 -- 172.21.15.45:0/3416917303 >> v1:172.21.15.45:6790/0 conn(0x7f0d8802a5a0 legacy=0x7f0d880c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.876+0000 7f0d94d86640 1 -- 172.21.15.45:0/3416917303 shutdown_connections 2024-03-29T10:56:43.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.876+0000 7f0d94d86640 1 -- 172.21.15.45:0/3416917303 >> 172.21.15.45:0/3416917303 conn(0x7f0d880058f0 msgr2=0x7f0d88005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:43.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.876+0000 7f0d94d86640 1 -- 172.21.15.45:0/3416917303 shutdown_connections 2024-03-29T10:56:43.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.877+0000 7f0d94d86640 1 -- 172.21.15.45:0/3416917303 wait complete. 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 10 monclient: build_initial_monmap 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 10 monclient: monmap: 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:43.861 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 1 librados: starting msgr at 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 1 librados: starting objecter 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 1 Processor -- start 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 1 -- start start 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 1 librados: setting wanted keys 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 1 librados: calling monclient init 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.878+0000 7f0d94d86640 10 monclient: init 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8800b8a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8802a5a0 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:43.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8802b4c0 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:43.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880d57a0 con 0x7f0d8802b4c0 2024-03-29T10:56:43.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880d6600 con 0x7f0d8802a5a0 2024-03-29T10:56:43.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880d5430 con 0x7f0d8800b8a0 2024-03-29T10:56:43.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:43.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.879+0000 7f0d94d86640 10 monclient(hunting): authenticate will time out at 1078.148738s 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.884+0000 7f0d8f7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0d8800b8a0 0x7f0d880d7210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43267/0 (socket says 172.21.15.45:43267) 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.884+0000 7f0d8f7fe640 1 -- 172.21.15.45:0/3137203371 learned_addr learned my addr 172.21.15.45:0/3137203371 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 client.?.objecter ms_handle_connect 0x7f0d8800b8a0 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621619560 0 0) 0x7f0d880d5430 con 0x7f0d8800b8a0 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:56:43.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d74004eb0 con 0x7f0d8800b8a0 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554430348 0 0) 0x7f0d74004eb0 con 0x7f0d8800b8a0 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d74005300 con 0x7f0d8800b8a0 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0d7c0019f0 con 0x7f0d8800b8a0 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): dump: 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:43.868 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:43.869 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:43.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.885+0000 7f0d7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:43.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.886+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1275148770 0 0) 0x7f0d74005300 con 0x7f0d8800b8a0 2024-03-29T10:56:43.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.886+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 >> v1:172.21.15.116:6789/0 conn(0x7f0d8802a5a0 legacy=0x7f0d880b1080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 >> v1:172.21.15.45:6789/0 conn(0x7f0d8802b4c0 legacy=0x7f0d8802d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 1 monclient: found mon.c 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880d7920 con 0x7f0d8800b8a0 2024-03-29T10:56:43.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 10 monclient: _finish_auth 0 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 10 monclient: _renew_subs 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d880d7b70 con 0x7f0d8800b8a0 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 10 monclient: _renew_subs 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d880b1970 con 0x7f0d8800b8a0 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.888+0000 7f0d94d86640 1 librados: init done 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.890+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d7c001cf0 con 0x7f0d8800b8a0 2024-03-29T10:56:43.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.890+0000 7f0d7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:43.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.890+0000 7f0d8dffb640 4 set_mon_vals no callback set 2024-03-29T10:56:43.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.890+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0d7c0019f0 con 0x7f0d8800b8a0 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.891+0000 7f0d7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.891+0000 7f0d7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.891+0000 7f0d7bfff640 10 monclient: dump: 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.891+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f0d7c003980 con 0x7f0d8800b8a0 2024-03-29T10:56:43.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.893+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f0d7c054730 con 0x7f0d8800b8a0 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.893+0000 7f0d7bfff640 10 client.4403.objecter ms_dispatch 0x7f0d90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.893+0000 7f0d7bfff640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.893+0000 7f0d7bfff640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.893+0000 7f0d7bfff640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.893+0000 7f0d8effd640 10 monclient: get_auth_request con 0x7f0d7406dfe0 auth_method 0 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.894+0000 7f0d94d86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.894+0000 7f0d94d86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.894+0000 7f0d94d86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.894+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d70005080 con 0x7f0d8800b8a0 2024-03-29T10:56:43.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.894+0000 7f0d7bfff640 10 client.4403.objecter ms_handle_connect 0x7f0d7406dfe0 2024-03-29T10:56:43.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.902+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d7c053e00 con 0x7f0d8800b8a0 2024-03-29T10:56:43.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.902+0000 7f0d7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:43.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:43.902+0000 7f0d7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.238+0000 7f0d94d86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.238+0000 7f0d94d86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.238+0000 7f0d94d86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.238+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d70005330 con 0x7f0d8800b8a0 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.239+0000 7f0d7bfff640 1 -- 172.21.15.45:0/3137203371 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f0d7c053e00 con 0x7f0d8800b8a0 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.239+0000 7f0d7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.239+0000 7f0d7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.243+0000 7f0d94d86640 10 librados: watch_flush enter 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.243+0000 7f0d94d86640 10 librados: watch_flush exit 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.243+0000 7f0d94d86640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.243+0000 7f0d94d86640 10 client.4403.objecter successfully canceled tick 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.245+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 >> v1:172.21.15.116:6816/3290579615 conn(0x7f0d7406dfe0 legacy=0x7f0d740704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.245+0000 7f0d94d86640 10 monclient: shutdown 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 >> v1:172.21.15.45:6790/0 conn(0x7f0d8800b8a0 legacy=0x7f0d880d7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 shutdown_connections 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 >> 172.21.15.45:0/3137203371 conn(0x7f0d880058f0 msgr2=0x7f0d88033f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 shutdown_connections 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 1 -- 172.21.15.45:0/3137203371 wait complete. 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:44.246+0000 7f0d94d86640 1 librados: shutdown 2024-03-29T10:56:44.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:51.055 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:51.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.224+0000 7f80a80c1640 10 monclient: get_monmap_and_config 2024-03-29T10:56:51.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.224+0000 7f80a80c1640 10 monclient: build_initial_monmap 2024-03-29T10:56:51.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.224+0000 7f80a80c1640 10 monclient: monmap: 2024-03-29T10:56:51.205 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:51.225620+0000 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:51.225620+0000 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:51.206 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:51.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.226+0000 7f80a80c1640 1 Processor -- start 2024-03-29T10:56:51.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.227+0000 7f80a80c1640 1 -- start start 2024-03-29T10:56:51.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.227+0000 7f80a80c1640 10 monclient: init 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient(hunting): picked mon.noname-a con 0x7f80980389e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient(hunting): picked mon.noname-b con 0x7f8098038dc0 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient(hunting): picked mon.noname-c con 0x7f8098030660 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80980c54c0 con 0x7f80980389e0 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8098036430 con 0x7f8098030660 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8098039840 con 0x7f8098038dc0 2024-03-29T10:56:51.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:51.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.228+0000 7f80a80c1640 10 monclient(hunting): authenticate will time out at 1085.497748s 2024-03-29T10:56:51.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.232+0000 7f80a565c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8098038dc0 0x7f80980cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57230/0 (socket says 172.21.15.45:57230) 2024-03-29T10:56:51.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.232+0000 7f80a565c640 1 -- 172.21.15.45:0/505718005 learned_addr learned my addr 172.21.15.45:0/505718005 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:51.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993481119 0 0) 0x7f8098039840 con 0x7f8098038dc0 2024-03-29T10:56:51.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:56:51.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f808c003520 con 0x7f8098038dc0 2024-03-29T10:56:51.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874253880 0 0) 0x7f80980c54c0 con 0x7f80980389e0 2024-03-29T10:56:51.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.233+0000 7f80a4e5b640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8098039840 con 0x7f80980389e0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 867289484 0 0) 0x7f8098039840 con 0x7f80980389e0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 >> v1:172.21.15.45:6790/0 conn(0x7f8098038dc0 legacy=0x7f80980cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 >> v1:172.21.15.116:6789/0 conn(0x7f8098030660 legacy=0x7f80980d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: _finish_hunting 0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 monclient: found mon.noname-a 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8098035690 con 0x7f80980389e0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: _finish_auth 0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: _check_auth_tickets 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80a004f730 con 0x7f80980389e0 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: dump: 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a80c1640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a80c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80a0075f10 con 0x7f80980389e0 2024-03-29T10:56:51.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:51.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 1 -- 172.21.15.45:0/505718005 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80a00758b0 con 0x7f80980389e0 2024-03-29T10:56:51.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:51.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.234+0000 7f80a4e5b640 10 monclient: dump: 2024-03-29T10:56:51.218 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:51.218 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.235+0000 7f80a80c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.235+0000 7f80a80c1640 4 set_mon_vals no callback set 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.236+0000 7f80a80c1640 10 monclient: shutdown 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.236+0000 7f80a80c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:51.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.236+0000 7f80a80c1640 1 -- 172.21.15.45:0/505718005 >> v1:172.21.15.45:6789/0 conn(0x7f80980389e0 legacy=0x7f8098033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.242+0000 7f80a80c1640 1 -- 172.21.15.45:0/505718005 shutdown_connections 2024-03-29T10:56:51.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.242+0000 7f80a80c1640 1 -- 172.21.15.45:0/505718005 >> 172.21.15.45:0/505718005 conn(0x7f8098005c00 msgr2=0x7f8098006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:51.223 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.242+0000 7f80a80c1640 1 -- 172.21.15.45:0/505718005 shutdown_connections 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.242+0000 7f80a80c1640 1 -- 172.21.15.45:0/505718005 wait complete. 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.243+0000 7f80a80c1640 10 monclient: build_initial_monmap 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.244+0000 7f80a80c1640 10 monclient: monmap: 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:51.225 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:51.226 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:51.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.244+0000 7f80a80c1640 1 librados: starting msgr at 2024-03-29T10:56:51.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.244+0000 7f80a80c1640 1 librados: starting objecter 2024-03-29T10:56:51.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.244+0000 7f80a80c1640 1 Processor -- start 2024-03-29T10:56:51.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 1 -- start start 2024-03-29T10:56:51.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 1 librados: setting wanted keys 2024-03-29T10:56:51.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 1 librados: calling monclient init 2024-03-29T10:56:51.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient: init 2024-03-29T10:56:51.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:51.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient(hunting): picked mon.noname-b con 0x7f8098030660 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient(hunting): picked mon.noname-a con 0x7f80980389e0 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient(hunting): picked mon.noname-c con 0x7f80980c2a20 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809802df70 con 0x7f80980389e0 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8098009950 con 0x7f8098030660 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8098009c30 con 0x7f80980c2a20 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.245+0000 7f80a80c1640 10 monclient(hunting): authenticate will time out at 1085.514893s 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.248+0000 7f80a665e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f80980c2a20 0x7f809802d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59726/0 (socket says 172.21.15.45:59726) 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.248+0000 7f80a665e640 1 -- 172.21.15.45:0/1136834719 learned_addr learned my addr 172.21.15.45:0/1136834719 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 10 client.?.objecter ms_handle_connect 0x7f80980c2a20 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478571635 0 0) 0x7f8098009c30 con 0x7f80980c2a20 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.249+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8084004eb0 con 0x7f80980c2a20 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 client.?.objecter ms_handle_connect 0x7f80980389e0 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278890927 0 0) 0x7f8084004eb0 con 0x7f80980c2a20 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8098009c30 con 0x7f80980c2a20 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f8090001950 con 0x7f80980c2a20 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): dump: 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:51.238 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161440707 0 0) 0x7f809802df70 con 0x7f80980389e0 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:56:51.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.250+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8084004eb0 con 0x7f80980389e0 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 591179671 0 0) 0x7f8098009c30 con 0x7f80980c2a20 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 >> v1:172.21.15.116:6789/0 conn(0x7f8098030660 legacy=0x7f8098164150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 >> v1:172.21.15.45:6789/0 conn(0x7f80980389e0 legacy=0x7f809802d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 1 monclient: found mon.c 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809802f100 con 0x7f80980c2a20 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 10 monclient: _renew_subs 2024-03-29T10:56:51.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8098164860 con 0x7f80980c2a20 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 10 monclient: _renew_subs 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80980c74f0 con 0x7f80980c2a20 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.251+0000 7f80a80c1640 1 librados: init done 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8090003610 con 0x7f80980c2a20 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80900039e0 con 0x7f80980c2a20 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80977fe640 4 set_mon_vals no callback set 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.252+0000 7f80967fc640 10 monclient: dump: 2024-03-29T10:56:51.241 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.253+0000 7f80a80c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.253+0000 7f80a80c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.253+0000 7f80a80c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.253+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8088005080 con 0x7f80980c2a20 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.255+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f8090003b40 con 0x7f80980c2a20 2024-03-29T10:56:51.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.255+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f8090084a40 con 0x7f80980c2a20 2024-03-29T10:56:51.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.255+0000 7f80967fc640 10 client.4409.objecter ms_dispatch 0x7f80a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:51.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.255+0000 7f80967fc640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:51.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.255+0000 7f80967fc640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:51.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.255+0000 7f80967fc640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T10:56:51.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.256+0000 7f80a5e5d640 10 monclient: get_auth_request con 0x7f808406dad0 auth_method 0 2024-03-29T10:56:51.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.257+0000 7f80967fc640 10 client.4409.objecter ms_handle_connect 0x7f808406dad0 2024-03-29T10:56:51.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.263+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8090050380 con 0x7f80980c2a20 2024-03-29T10:56:51.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.263+0000 7f80967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:51.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.263+0000 7f80967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:51.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.597+0000 7f80a80c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:51.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.597+0000 7f80a80c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:51.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.597+0000 7f80a80c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:51.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.597+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8088005330 con 0x7f80980c2a20 2024-03-29T10:56:51.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.598+0000 7f80967fc640 1 -- 172.21.15.45:0/1136834719 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f80900840f0 con 0x7f80980c2a20 2024-03-29T10:56:51.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.598+0000 7f80967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:51.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.598+0000 7f80967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:51.580 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:51.580 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:51.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.602+0000 7f80a80c1640 10 librados: watch_flush enter 2024-03-29T10:56:51.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 10 librados: watch_flush exit 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 10 client.4409.objecter successfully canceled tick 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 >> v1:172.21.15.116:6816/3290579615 conn(0x7f808406dad0 legacy=0x7f808406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 10 monclient: shutdown 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.603+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 >> v1:172.21.15.45:6790/0 conn(0x7f80980c2a20 legacy=0x7f809802d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.604+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 shutdown_connections 2024-03-29T10:56:51.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.604+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 >> 172.21.15.45:0/1136834719 conn(0x7f8098005c00 msgr2=0x7f80980c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:51.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.604+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 shutdown_connections 2024-03-29T10:56:51.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.604+0000 7f80a80c1640 1 -- 172.21.15.45:0/1136834719 wait complete. 2024-03-29T10:56:51.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:51.604+0000 7f80a80c1640 1 librados: shutdown 2024-03-29T10:56:51.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:58.410 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.576+0000 7f84c09fe640 10 monclient: get_monmap_and_config 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.576+0000 7f84c09fe640 10 monclient: build_initial_monmap 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.576+0000 7f84c09fe640 10 monclient: monmap: 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:56:58.577522+0000 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:56:58.577522+0000 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:58.557 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.577+0000 7f84c09fe640 1 Processor -- start 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.577+0000 7f84c09fe640 1 -- start start 2024-03-29T10:56:58.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.577+0000 7f84c09fe640 10 monclient: init 2024-03-29T10:56:58.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient(hunting): picked mon.noname-b con 0x7f84bc0af290 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient(hunting): picked mon.noname-a con 0x7f84bc0ae740 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient(hunting): picked mon.noname-c con 0x7f84bc0af670 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0b0980 con 0x7f84bc0ae740 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0b39f0 con 0x7f84bc0af670 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0aebd0 con 0x7f84bc0af290 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:58.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.578+0000 7f84c09fe640 10 monclient(hunting): authenticate will time out at 1092.847908s 2024-03-29T10:56:58.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.580+0000 7f84ba59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f84bc0af290 0x7f84bc0b4430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55902/0 (socket says 172.21.15.45:55902) 2024-03-29T10:56:58.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.580+0000 7f84ba59c640 1 -- 172.21.15.45:0/425278775 learned_addr learned my addr 172.21.15.45:0/425278775 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:58.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279235813 0 0) 0x7f84bc0aebd0 con 0x7f84bc0af290 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a4003520 con 0x7f84bc0af290 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270172973 0 0) 0x7f84bc0b39f0 con 0x7f84bc0af670 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.582+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc0aebd0 con 0x7f84bc0af670 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157075050 0 0) 0x7f84bc0aebd0 con 0x7f84bc0af670 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 >> v1:172.21.15.45:6790/0 conn(0x7f84bc0af290 legacy=0x7f84bc0b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 >> v1:172.21.15.45:6789/0 conn(0x7f84bc0ae740 legacy=0x7f84bc1439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 monclient: found mon.noname-c 2024-03-29T10:56:58.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0b4b40 con 0x7f84bc0af670 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: _finish_auth 0 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f84b40018a0 con 0x7f84bc0af670 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: dump: 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.566 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84c09fe640 5 monclient: authenticate success, global_id 4411 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84c09fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84b4001a00 con 0x7f84bc0af670 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 1 -- 172.21.15.45:0/425278775 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f84b4001e50 con 0x7f84bc0af670 2024-03-29T10:56:58.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84b959a640 10 monclient: dump: 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:58.568 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84c09fe640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.584+0000 7f84c09fe640 4 set_mon_vals no callback set 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.585+0000 7f84c09fe640 10 monclient: shutdown 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.585+0000 7f84c09fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.585+0000 7f84c09fe640 1 -- 172.21.15.45:0/425278775 >> v1:172.21.15.116:6789/0 conn(0x7f84bc0af670 legacy=0x7f84bc151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.589+0000 7f84c09fe640 1 -- 172.21.15.45:0/425278775 shutdown_connections 2024-03-29T10:56:58.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.589+0000 7f84c09fe640 1 -- 172.21.15.45:0/425278775 >> 172.21.15.45:0/425278775 conn(0x7f84bc066a00 msgr2=0x7f84bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:58.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.592+0000 7f84c09fe640 1 -- 172.21.15.45:0/425278775 shutdown_connections 2024-03-29T10:56:58.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.592+0000 7f84c09fe640 1 -- 172.21.15.45:0/425278775 wait complete. 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.593+0000 7f84c09fe640 10 monclient: build_initial_monmap 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.593+0000 7f84c09fe640 10 monclient: monmap: 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:58.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.593+0000 7f84c09fe640 1 librados: starting msgr at 2024-03-29T10:56:58.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.593+0000 7f84c09fe640 1 librados: starting objecter 2024-03-29T10:56:58.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.593+0000 7f84c09fe640 1 Processor -- start 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.594+0000 7f84c09fe640 1 -- start start 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.594+0000 7f84c09fe640 1 librados: setting wanted keys 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.594+0000 7f84c09fe640 1 librados: calling monclient init 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.594+0000 7f84c09fe640 10 monclient: init 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.595+0000 7f84c09fe640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.595+0000 7f84c09fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:58.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.595+0000 7f84c09fe640 10 monclient(hunting): picked mon.noname-b con 0x7f84bc0af290 addr v1:172.21.15.116:6789/0 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.595+0000 7f84c09fe640 10 monclient(hunting): picked mon.noname-c con 0x7f84bc0af670 addr v1:172.21.15.45:6790/0 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.595+0000 7f84b9d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f84bc0af670 0x7f84bc1e2af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57352/0 (socket says 172.21.15.45:57352) 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84c09fe640 10 monclient(hunting): picked mon.noname-a con 0x7f84bc1d02d0 addr v1:172.21.15.45:6789/0 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84c09fe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc1d1410 con 0x7f84bc1d02d0 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84c09fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc1e3910 con 0x7f84bc0af290 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84c09fe640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc1d0f30 con 0x7f84bc0af670 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84c09fe640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:58.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84c09fe640 10 monclient(hunting): authenticate will time out at 1092.867076s 2024-03-29T10:56:58.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84b9d9b640 1 -- 172.21.15.45:0/2685494493 learned_addr learned my addr 172.21.15.45:0/2685494493 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:56:58.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 10 client.?.objecter ms_handle_connect 0x7f84bc0af670 2024-03-29T10:56:58.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:58.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797897932 0 0) 0x7f84bc1d0f30 con 0x7f84bc0af670 2024-03-29T10:56:58.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f849c004eb0 con 0x7f84bc0af670 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1523353510 0 0) 0x7f849c004eb0 con 0x7f84bc0af670 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.598+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84bc1d0f30 con 0x7f84bc0af670 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f84b0001960 con 0x7f84bc0af670 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 10 monclient(hunting): dump: 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.580 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 57195722 0 0) 0x7f84bc1d0f30 con 0x7f84bc0af670 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.599+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 >> v1:172.21.15.116:6789/0 conn(0x7f84bc0af290 legacy=0x7f84bc1e7180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:58.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.600+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 >> v1:172.21.15.45:6789/0 conn(0x7f84bc1d02d0 legacy=0x7f84bc1e3200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 1 monclient: found mon.c 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc1d3340 con 0x7f84bc0af670 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:58.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84c09fe640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84c09fe640 10 monclient: _renew_subs 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84c09fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.602+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc1d1f50 con 0x7f84bc0af670 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84c09fe640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84c09fe640 10 monclient: _renew_subs 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84c09fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc142a20 con 0x7f84bc0af670 2024-03-29T10:56:58.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84c09fe640 1 librados: init done 2024-03-29T10:56:58.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84b0001c80 con 0x7f84bc0af670 2024-03-29T10:56:58.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:58.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.603+0000 7f84b8d99640 4 set_mon_vals no callback set 2024-03-29T10:56:58.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f84b0003a40 con 0x7f84bc0af670 2024-03-29T10:56:58.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84aaffd640 10 monclient: dump: 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84c09fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84c09fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84c09fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.604+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a0005080 con 0x7f84bc0af670 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.605+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f84b0003ba0 con 0x7f84bc0af670 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.606+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f84b0084af0 con 0x7f84bc0af670 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.606+0000 7f84aaffd640 10 client.4415.objecter ms_dispatch 0x7f84bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.606+0000 7f84aaffd640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.606+0000 7f84aaffd640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:58.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.606+0000 7f84aaffd640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T10:56:58.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.606+0000 7f84ba59c640 10 monclient: get_auth_request con 0x7f849c06db60 auth_method 0 2024-03-29T10:56:58.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.608+0000 7f84aaffd640 10 client.4415.objecter ms_handle_connect 0x7f849c06db60 2024-03-29T10:56:58.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.615+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84b00503d0 con 0x7f84bc0af670 2024-03-29T10:56:58.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.615+0000 7f84aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.615+0000 7f84aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:58.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.947+0000 7f84c09fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:58.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.947+0000 7f84c09fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:58.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.947+0000 7f84c09fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:56:58.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.947+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84a0005330 con 0x7f84bc0af670 2024-03-29T10:56:58.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.950+0000 7f84aaffd640 1 -- 172.21.15.45:0/2685494493 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f84b0052300 con 0x7f84bc0af670 2024-03-29T10:56:58.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.950+0000 7f84aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:58.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.950+0000 7f84aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:58.932 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:56:58.932 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:58.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.954+0000 7f84c09fe640 10 librados: watch_flush enter 2024-03-29T10:56:58.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.954+0000 7f84c09fe640 10 librados: watch_flush exit 2024-03-29T10:56:58.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.954+0000 7f84c09fe640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T10:56:58.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.954+0000 7f84c09fe640 10 client.4415.objecter successfully canceled tick 2024-03-29T10:56:58.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.954+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 >> v1:172.21.15.116:6816/3290579615 conn(0x7f849c06db60 legacy=0x7f849c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.955+0000 7f84c09fe640 10 monclient: shutdown 2024-03-29T10:56:58.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.955+0000 7f84c09fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:58.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.955+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 >> v1:172.21.15.45:6790/0 conn(0x7f84bc0af670 legacy=0x7f84bc1e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.956+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 shutdown_connections 2024-03-29T10:56:58.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.956+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 >> 172.21.15.45:0/2685494493 conn(0x7f84bc066a00 msgr2=0x7f84bc1da680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:58.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.956+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 shutdown_connections 2024-03-29T10:56:58.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.956+0000 7f84c09fe640 1 -- 172.21.15.45:0/2685494493 wait complete. 2024-03-29T10:56:58.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:56:58.956+0000 7f84c09fe640 1 librados: shutdown 2024-03-29T10:56:58.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:05.761 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:05.912 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.932+0000 7fb0f167b640 10 monclient: get_monmap_and_config 2024-03-29T10:57:05.912 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.932+0000 7fb0f167b640 10 monclient: build_initial_monmap 2024-03-29T10:57:05.912 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.932+0000 7fb0f167b640 10 monclient: monmap: 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:05.932981+0000 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:05.932981+0000 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:05.913 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.933+0000 7fb0f167b640 1 Processor -- start 2024-03-29T10:57:05.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.933+0000 7fb0f167b640 1 -- start start 2024-03-29T10:57:05.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.933+0000 7fb0f167b640 10 monclient: init 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ec1457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ec13f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ec0a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0aaef0 con 0x7fb0ec13f650 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec145bd0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec145eb0 con 0x7fb0ec1457f0 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:05.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.934+0000 7fb0f167b640 10 monclient(hunting): authenticate will time out at 1100.203661s 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.937+0000 7fb0eb7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb0ec0a9d70 0x7fb0ec14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:43662/0 (socket says 172.21.15.45:43662) 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.937+0000 7fb0eb7fe640 1 -- 172.21.15.45:0/1609593347 learned_addr learned my addr 172.21.15.45:0/1609593347 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808801617 0 0) 0x7fb0ec145bd0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d4003520 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540479441 0 0) 0x7fb0ec145eb0 con 0x7fb0ec1457f0 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec145bd0 con 0x7fb0ec1457f0 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549131760 0 0) 0x7fb0d4003520 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 >> v1:172.21.15.45:6790/0 conn(0x7fb0ec1457f0 legacy=0x7fb0ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 >> v1:172.21.15.45:6789/0 conn(0x7fb0ec13f650 legacy=0x7fb0ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 monclient: found mon.noname-c 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec146340 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e0001690 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: dump: 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0f167b640 5 monclient: authenticate success, global_id 4414 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0f167b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:05.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e00017f0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 1 -- 172.21.15.45:0/1609593347 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e0001c00 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0e9ffb640 10 monclient: dump: 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:05.924 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0f167b640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:05.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.938+0000 7fb0f167b640 4 set_mon_vals no callback set 2024-03-29T10:57:05.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.939+0000 7fb0f167b640 10 monclient: shutdown 2024-03-29T10:57:05.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.939+0000 7fb0f167b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:05.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.939+0000 7fb0f167b640 1 -- 172.21.15.45:0/1609593347 >> v1:172.21.15.116:6789/0 conn(0x7fb0ec0a9d70 legacy=0x7fb0ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.942+0000 7fb0f167b640 1 -- 172.21.15.45:0/1609593347 shutdown_connections 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.942+0000 7fb0f167b640 1 -- 172.21.15.45:0/1609593347 >> 172.21.15.45:0/1609593347 conn(0x7fb0ec066a00 msgr2=0x7fb0ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.942+0000 7fb0f167b640 1 -- 172.21.15.45:0/1609593347 shutdown_connections 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.942+0000 7fb0f167b640 1 -- 172.21.15.45:0/1609593347 wait complete. 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.943+0000 7fb0f167b640 10 monclient: build_initial_monmap 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.943+0000 7fb0f167b640 10 monclient: monmap: 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:05.927 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.943+0000 7fb0f167b640 1 librados: starting msgr at 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.943+0000 7fb0f167b640 1 librados: starting objecter 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.944+0000 7fb0f167b640 1 Processor -- start 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.944+0000 7fb0f167b640 1 -- start start 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.944+0000 7fb0f167b640 1 librados: setting wanted keys 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.944+0000 7fb0f167b640 1 librados: calling monclient init 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.944+0000 7fb0f167b640 10 monclient: init 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.945+0000 7fb0f167b640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.945+0000 7fb0f167b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:05.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.946+0000 7fb0f167b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ec0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.948+0000 7fb0f167b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ec13f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0f167b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ec1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0f167b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec141fc0 con 0x7fb0ec13f650 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0f167b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec136780 con 0x7fb0ec1457f0 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0f167b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec06eab0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0f167b640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0f167b640 10 monclient(hunting): authenticate will time out at 1100.219201s 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0eaffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb0ec0a9d70 0x7fb0ec142500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57403/0 (socket says 172.21.15.45:57403) 2024-03-29T10:57:05.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.950+0000 7fb0eaffd640 1 -- 172.21.15.45:0/814875024 learned_addr learned my addr 172.21.15.45:0/814875024 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:05.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 client.?.objecter ms_handle_connect 0x7fb0ec0a9d70 2024-03-29T10:57:05.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:05.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 client.?.objecter ms_handle_connect 0x7fb0ec13f650 2024-03-29T10:57:05.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:05.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245813278 0 0) 0x7fb0ec06eab0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d8004eb0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442044377 0 0) 0x7fb0ec141fc0 con 0x7fb0ec13f650 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.951+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec06eab0 con 0x7fb0ec13f650 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3480971207 0 0) 0x7fb0d8004eb0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0ec141fc0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0dc001cf0 con 0x7fb0ec0a9d70 2024-03-29T10:57:05.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): dump: 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2466297276 0 0) 0x7fb0ec06eab0 con 0x7fb0ec13f650 2024-03-29T10:57:05.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0d8004eb0 con 0x7fb0ec13f650 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e4001830 con 0x7fb0ec13f650 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient(hunting): dump: 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 878513653 0 0) 0x7fb0d8004eb0 con 0x7fb0ec13f650 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 >> v1:172.21.15.45:6790/0 conn(0x7fb0ec0a9d70 legacy=0x7fb0ec142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 >> v1:172.21.15.116:6789/0 conn(0x7fb0ec1457f0 legacy=0x7fb0ec136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 monclient: found mon.a 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec06ed90 con 0x7fb0ec13f650 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0f167b640 5 monclient: authenticate success, global_id 4392 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0f167b640 10 monclient: _renew_subs 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0f167b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.952+0000 7fb0f167b640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec13fda0 con 0x7fb0ec13f650 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.953+0000 7fb0f167b640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:05.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.953+0000 7fb0f167b640 10 monclient: _renew_subs 2024-03-29T10:57:05.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.953+0000 7fb0f167b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:05.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.953+0000 7fb0f167b640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec142df0 con 0x7fb0ec13f650 2024-03-29T10:57:05.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.953+0000 7fb0f167b640 1 librados: init done 2024-03-29T10:57:05.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.955+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4003610 con 0x7fb0ec13f650 2024-03-29T10:57:05.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.955+0000 7fb0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.955+0000 7fb0e97fa640 4 set_mon_vals no callback set 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.955+0000 7fb0d17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.956+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e4003a20 con 0x7fb0ec13f650 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.956+0000 7fb0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.956+0000 7fb0d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.956+0000 7fb0d37fe640 10 monclient: dump: 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:05.940 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.956+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fb0e4003c40 con 0x7fb0ec13f650 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc005080 con 0x7fb0ec13f650 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fb0e4083770 con 0x7fb0ec13f650 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d37fe640 10 client.4392.objecter ms_dispatch 0x7fb0ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d37fe640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d37fe640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0d37fe640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.957+0000 7fb0eaffd640 10 monclient: get_auth_request con 0x7fb0d806dff0 auth_method 0 2024-03-29T10:57:05.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.958+0000 7fb0d37fe640 10 client.4392.objecter ms_handle_connect 0x7fb0d806dff0 2024-03-29T10:57:05.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.966+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0e404f110 con 0x7fb0ec13f650 2024-03-29T10:57:05.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.966+0000 7fb0d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:05.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:05.966+0000 7fb0d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.304+0000 7fb0d17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.304+0000 7fb0d17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.304+0000 7fb0d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.304+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0cc005330 con 0x7fb0ec13f650 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.308+0000 7fb0d37fe640 1 -- 172.21.15.45:0/814875024 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fb0e4051040 con 0x7fb0ec13f650 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.308+0000 7fb0d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.308+0000 7fb0d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 10 librados: watch_flush enter 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 10 librados: watch_flush exit 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 10 client.4392.objecter successfully canceled tick 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 >> v1:172.21.15.116:6816/3290579615 conn(0x7fb0d806dff0 legacy=0x7fb0d80704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 10 monclient: shutdown 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:06.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.312+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 >> v1:172.21.15.45:6789/0 conn(0x7fb0ec13f650 legacy=0x7fb0ec135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.313+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 shutdown_connections 2024-03-29T10:57:06.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.313+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 >> 172.21.15.45:0/814875024 conn(0x7fb0ec066a00 msgr2=0x7fb0ec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:06.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.313+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 shutdown_connections 2024-03-29T10:57:06.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.313+0000 7fb0d17fa640 1 -- 172.21.15.45:0/814875024 wait complete. 2024-03-29T10:57:06.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:06.313+0000 7fb0d17fa640 1 librados: shutdown 2024-03-29T10:57:06.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:13.119 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.292+0000 7fdc4f8fb640 10 monclient: get_monmap_and_config 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.292+0000 7fdc4f8fb640 10 monclient: build_initial_monmap 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.292+0000 7fdc4f8fb640 10 monclient: monmap: 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:13.293434+0000 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:13.293434+0000 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:13.273 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.274 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:13.274 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:13.274 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:13.274 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.293+0000 7fdc4f8fb640 1 Processor -- start 2024-03-29T10:57:13.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.294+0000 7fdc4f8fb640 1 -- start start 2024-03-29T10:57:13.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.294+0000 7fdc4f8fb640 10 monclient: init 2024-03-29T10:57:13.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.294+0000 7fdc4f8fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:13.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.294+0000 7fdc4f8fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:13.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.294+0000 7fdc4f8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdc481457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:13.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.294+0000 7fdc4f8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdc4813f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4f8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdc480a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4f8fb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480aaef0 con 0x7fdc481457f0 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4f8fb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48145bd0 con 0x7fdc4813f650 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4f8fb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48145eb0 con 0x7fdc480a9d70 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4f8fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4f8fb640 10 monclient(hunting): authenticate will time out at 1107.567499s 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4de98640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdc480a9d70 0x7fdc4814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37583/0 (socket says 172.21.15.45:37583) 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc4de98640 1 -- 172.21.15.45:0/3319270911 learned_addr learned my addr 172.21.15.45:0/3319270911 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062566225 0 0) 0x7fdc48145eb0 con 0x7fdc480a9d70 2024-03-29T10:57:13.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc3ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc3ffff640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.298+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc34003520 con 0x7fdc480a9d70 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.299+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588515046 0 0) 0x7fdc34003520 con 0x7fdc480a9d70 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.299+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 >> v1:172.21.15.116:6789/0 conn(0x7fdc4813f650 legacy=0x7fdc480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:13.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.299+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 >> v1:172.21.15.45:6789/0 conn(0x7fdc481457f0 legacy=0x7fdc4813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 1 monclient: found mon.noname-b 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48146340 con 0x7fdc480a9d70 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: _finish_auth 0 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:13.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdc38001690 con 0x7fdc480a9d70 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.301+0000 7fdc3ffff640 10 monclient: dump: 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc4f8fb640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc4f8fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc38001860 con 0x7fdc480a9d70 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc3ffff640 1 -- 172.21.15.45:0/3319270911 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdc38001cb0 con 0x7fdc480a9d70 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc3ffff640 10 monclient: dump: 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:13.285 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:13.286 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc4f8fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.302+0000 7fdc4f8fb640 4 set_mon_vals no callback set 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.303+0000 7fdc4f8fb640 10 monclient: shutdown 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.303+0000 7fdc4f8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.303+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/3319270911 >> v1:172.21.15.45:6790/0 conn(0x7fdc480a9d70 legacy=0x7fdc4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.303+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/3319270911 shutdown_connections 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.303+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/3319270911 >> 172.21.15.45:0/3319270911 conn(0x7fdc48066a00 msgr2=0x7fdc48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.304+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/3319270911 shutdown_connections 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.304+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/3319270911 wait complete. 2024-03-29T10:57:13.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.306+0000 7fdc4f8fb640 10 monclient: build_initial_monmap 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.306+0000 7fdc4f8fb640 10 monclient: monmap: 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.306+0000 7fdc4f8fb640 1 librados: starting msgr at 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.306+0000 7fdc4f8fb640 1 librados: starting objecter 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.307+0000 7fdc4f8fb640 1 Processor -- start 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.307+0000 7fdc4f8fb640 1 -- start start 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.307+0000 7fdc4f8fb640 1 librados: setting wanted keys 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.307+0000 7fdc4f8fb640 1 librados: calling monclient init 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.307+0000 7fdc4f8fb640 10 monclient: init 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.308+0000 7fdc4f8fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:13.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.308+0000 7fdc4f8fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:13.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.312+0000 7fdc4f8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdc480a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:13.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.313+0000 7fdc4d697640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdc480a9d70 0x7fdc481361b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:46531/0 (socket says 172.21.15.45:46531) 2024-03-29T10:57:13.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.313+0000 7fdc4d697640 1 -- 172.21.15.45:0/1867278286 learned_addr learned my addr 172.21.15.45:0/1867278286 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:13.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.313+0000 7fdc4f8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdc4813f650 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:13.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.315+0000 7fdc3dffb640 10 client.?.objecter ms_handle_connect 0x7fdc480a9d70 2024-03-29T10:57:13.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.315+0000 7fdc3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:13.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc4f8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdc481457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:13.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48134300 con 0x7fdc481457f0 2024-03-29T10:57:13.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48133730 con 0x7fdc480a9d70 2024-03-29T10:57:13.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc481338c0 con 0x7fdc4813f650 2024-03-29T10:57:13.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc4f8fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:13.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc4f8fb640 10 monclient(hunting): authenticate will time out at 1107.585718s 2024-03-29T10:57:13.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc3dffb640 10 client.?.objecter ms_handle_connect 0x7fdc481457f0 2024-03-29T10:57:13.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:13.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.316+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124447922 0 0) 0x7fdc48134300 con 0x7fdc481457f0 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc2c004eb0 con 0x7fdc481457f0 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358008509 0 0) 0x7fdc48133730 con 0x7fdc480a9d70 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:57:13.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.317+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc48134300 con 0x7fdc480a9d70 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 193886884 0 0) 0x7fdc2c004eb0 con 0x7fdc481457f0 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc48133730 con 0x7fdc481457f0 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdc38001b70 con 0x7fdc481457f0 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): dump: 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.301 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 734604057 0 0) 0x7fdc48134300 con 0x7fdc480a9d70 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc2c004eb0 con 0x7fdc480a9d70 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdc44001710 con 0x7fdc480a9d70 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 10 monclient(hunting): dump: 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:13.302 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 647068264 0 0) 0x7fdc48133730 con 0x7fdc481457f0 2024-03-29T10:57:13.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.318+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 >> v1:172.21.15.45:6790/0 conn(0x7fdc4813f650 legacy=0x7fdc481d7160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:13.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 >> v1:172.21.15.116:6789/0 conn(0x7fdc480a9d70 legacy=0x7fdc481361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:13.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 1 monclient: found mon.a 2024-03-29T10:57:13.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:13.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:13.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48133a50 con 0x7fdc481457f0 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc38001010 con 0x7fdc481457f0 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fdc38004d90 con 0x7fdc481457f0 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc3dffb640 10 monclient: dump: 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.305 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc4f8fb640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc4f8fb640 10 monclient: _renew_subs 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.323+0000 7fdc4f8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc481424c0 con 0x7fdc481457f0 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc3f7fe640 4 set_mon_vals no callback set 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc4f8fb640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:13.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc4f8fb640 10 monclient: _renew_subs 2024-03-29T10:57:13.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc4f8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:13.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48134790 con 0x7fdc481457f0 2024-03-29T10:57:13.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.324+0000 7fdc4f8fb640 1 librados: init done 2024-03-29T10:57:13.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.327+0000 7fdc4f8fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.327+0000 7fdc4f8fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.327+0000 7fdc4f8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:13.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.327+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc48075f10 con 0x7fdc481457f0 2024-03-29T10:57:13.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.328+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fdc3801ad60 con 0x7fdc481457f0 2024-03-29T10:57:13.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc4d697640 10 monclient: get_auth_request con 0x7fdc2c06de00 auth_method 0 2024-03-29T10:57:13.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fdc38084b20 con 0x7fdc481457f0 2024-03-29T10:57:13.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc3dffb640 10 client.4395.objecter ms_dispatch 0x7fdc48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:13.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc3dffb640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:13.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc3dffb640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:13.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc3dffb640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T10:57:13.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.329+0000 7fdc3dffb640 10 client.4395.objecter ms_handle_connect 0x7fdc2c06de00 2024-03-29T10:57:13.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.336+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc38050400 con 0x7fdc481457f0 2024-03-29T10:57:13.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.336+0000 7fdc3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.336+0000 7fdc3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.667+0000 7fdc4f8fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.667+0000 7fdc4f8fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.667+0000 7fdc4f8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.667+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc480b3680 con 0x7fdc481457f0 2024-03-29T10:57:13.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.671+0000 7fdc3dffb640 1 -- 172.21.15.45:0/1867278286 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fdc38052330 con 0x7fdc481457f0 2024-03-29T10:57:13.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.671+0000 7fdc3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.671+0000 7fdc3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:13.653 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:13.653 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.675+0000 7fdc4f8fb640 10 librados: watch_flush enter 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.676+0000 7fdc4f8fb640 10 librados: watch_flush exit 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.676+0000 7fdc4f8fb640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.676+0000 7fdc4f8fb640 10 client.4395.objecter successfully canceled tick 2024-03-29T10:57:13.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.677+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 >> v1:172.21.15.116:6816/3290579615 conn(0x7fdc2c06de00 legacy=0x7fdc2c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.677+0000 7fdc4f8fb640 10 monclient: shutdown 2024-03-29T10:57:13.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.677+0000 7fdc4f8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:13.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.677+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 >> v1:172.21.15.45:6789/0 conn(0x7fdc481457f0 legacy=0x7fdc481e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.677+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 shutdown_connections 2024-03-29T10:57:13.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.677+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 >> 172.21.15.45:0/1867278286 conn(0x7fdc48066a00 msgr2=0x7fdc480b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.680+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 shutdown_connections 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.680+0000 7fdc4f8fb640 1 -- 172.21.15.45:0/1867278286 wait complete. 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:13.680+0000 7fdc4f8fb640 1 librados: shutdown 2024-03-29T10:57:13.685 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:20.492 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:20.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.656+0000 7efc8ce7d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:20.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.656+0000 7efc8ce7d640 10 monclient: build_initial_monmap 2024-03-29T10:57:20.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.656+0000 7efc8ce7d640 10 monclient: monmap: 2024-03-29T10:57:20.637 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:20.657698+0000 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:20.657698+0000 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:20.638 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:20.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.658+0000 7efc8ce7d640 1 Processor -- start 2024-03-29T10:57:20.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.658+0000 7efc8ce7d640 1 -- start start 2024-03-29T10:57:20.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.658+0000 7efc8ce7d640 10 monclient: init 2024-03-29T10:57:20.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:20.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient(hunting): picked mon.noname-c con 0x7efc880b0770 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient(hunting): picked mon.noname-a con 0x7efc8814a700 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient(hunting): picked mon.noname-b con 0x7efc880ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880aba70 con 0x7efc8814a700 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880aecb0 con 0x7efc880b0770 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880af150 con 0x7efc880ac5a0 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.659+0000 7efc8ce7d640 10 monclient(hunting): authenticate will time out at 1114.928878s 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.661+0000 7efc8659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efc880b0770 0x7efc88149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:53100/0 (socket says 172.21.15.45:53100) 2024-03-29T10:57:20.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.661+0000 7efc8659c640 1 -- 172.21.15.45:0/2942995103 learned_addr learned my addr 172.21.15.45:0/2942995103 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373307757 0 0) 0x7efc880aecb0 con 0x7efc880b0770 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc70003520 con 0x7efc880b0770 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353078039 0 0) 0x7efc880af150 con 0x7efc880ac5a0 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.664+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc880aecb0 con 0x7efc880ac5a0 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707573159 0 0) 0x7efc70003520 con 0x7efc880b0770 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 >> v1:172.21.15.45:6790/0 conn(0x7efc880ac5a0 legacy=0x7efc88151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 >> v1:172.21.15.45:6789/0 conn(0x7efc8814a700 legacy=0x7efc880abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 monclient: found mon.noname-c 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc880ac9d0 con 0x7efc880b0770 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: _finish_auth 0 2024-03-29T10:57:20.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efc78001670 con 0x7efc880b0770 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: dump: 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:20.648 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc78001900 con 0x7efc880b0770 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 1 -- 172.21.15.45:0/2942995103 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efc78001c90 con 0x7efc880b0770 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.665+0000 7efc8559a640 10 monclient: dump: 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.649 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.650 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.650 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:20.650 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:20.650 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:20.650 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:20.650 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.667+0000 7efc8ce7d640 5 monclient: authenticate success, global_id 4420 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.667+0000 7efc8ce7d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.667+0000 7efc8ce7d640 4 set_mon_vals no callback set 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.668+0000 7efc8ce7d640 10 monclient: shutdown 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.668+0000 7efc8ce7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.668+0000 7efc8ce7d640 1 -- 172.21.15.45:0/2942995103 >> v1:172.21.15.116:6789/0 conn(0x7efc880b0770 legacy=0x7efc88149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.670+0000 7efc8ce7d640 1 -- 172.21.15.45:0/2942995103 shutdown_connections 2024-03-29T10:57:20.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.670+0000 7efc8ce7d640 1 -- 172.21.15.45:0/2942995103 >> 172.21.15.45:0/2942995103 conn(0x7efc8806b660 msgr2=0x7efc8806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:20.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.672+0000 7efc8ce7d640 1 -- 172.21.15.45:0/2942995103 shutdown_connections 2024-03-29T10:57:20.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.672+0000 7efc8ce7d640 1 -- 172.21.15.45:0/2942995103 wait complete. 2024-03-29T10:57:20.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.674+0000 7efc8ce7d640 10 monclient: build_initial_monmap 2024-03-29T10:57:20.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.674+0000 7efc8ce7d640 10 monclient: monmap: 2024-03-29T10:57:20.654 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:20.654 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:20.654 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:20.655 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.674+0000 7efc8ce7d640 1 librados: starting msgr at 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.674+0000 7efc8ce7d640 1 librados: starting objecter 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.674+0000 7efc8ce7d640 1 Processor -- start 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.675+0000 7efc8ce7d640 1 -- start start 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.675+0000 7efc8ce7d640 1 librados: setting wanted keys 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.675+0000 7efc8ce7d640 1 librados: calling monclient init 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.675+0000 7efc8ce7d640 10 monclient: init 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.676+0000 7efc8ce7d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.676+0000 7efc8ce7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.676+0000 7efc8ce7d640 10 monclient(hunting): picked mon.noname-a con 0x7efc880ac5a0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:20.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.676+0000 7efc8ce7d640 10 monclient(hunting): picked mon.noname-c con 0x7efc880b0770 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:20.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.676+0000 7efc8ce7d640 10 monclient(hunting): picked mon.noname-b con 0x7efc8814a700 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:20.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc8ce7d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc88128570 con 0x7efc880ac5a0 2024-03-29T10:57:20.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc8ce7d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc881db640 con 0x7efc8814a700 2024-03-29T10:57:20.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc8ce7d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc881239d0 con 0x7efc880b0770 2024-03-29T10:57:20.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc8ce7d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:20.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc8ce7d640 10 monclient(hunting): authenticate will time out at 1114.947195s 2024-03-29T10:57:20.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc86d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efc8814a700 0x7efc881e6130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:43454/0 (socket says 172.21.15.45:43454) 2024-03-29T10:57:20.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.678+0000 7efc86d9d640 1 -- 172.21.15.45:0/1316884 learned_addr learned my addr 172.21.15.45:0/1316884 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 10 client.?.objecter ms_handle_connect 0x7efc8814a700 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059683134 0 0) 0x7efc881db640 con 0x7efc8814a700 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:57:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.679+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68004eb0 con 0x7efc8814a700 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963712434 0 0) 0x7efc68004eb0 con 0x7efc8814a700 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc881db640 con 0x7efc8814a700 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efc80001c40 con 0x7efc8814a700 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.661 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 10 client.?.objecter ms_handle_connect 0x7efc880ac5a0 2024-03-29T10:57:20.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.680+0000 7efc76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474749683 0 0) 0x7efc88128570 con 0x7efc880ac5a0 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68004eb0 con 0x7efc880ac5a0 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2650704851 0 0) 0x7efc881db640 con 0x7efc8814a700 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.681+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 >> v1:172.21.15.45:6790/0 conn(0x7efc880b0770 legacy=0x7efc881dbb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 >> v1:172.21.15.45:6789/0 conn(0x7efc880ac5a0 legacy=0x7efc88126110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 1 monclient: found mon.b 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:20.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc88123b60 con 0x7efc8814a700 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 10 monclient: _renew_subs 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc881dc270 con 0x7efc8814a700 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 10 monclient: _renew_subs 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc88127160 con 0x7efc8814a700 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc8ce7d640 1 librados: init done 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc80003990 con 0x7efc8814a700 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efc80003d60 con 0x7efc8814a700 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc76ffd640 10 monclient: dump: 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:20.669 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.683+0000 7efc77fff640 4 set_mon_vals no callback set 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.685+0000 7efc8ce7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.685+0000 7efc8ce7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.685+0000 7efc8ce7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:20.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.685+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6c005080 con 0x7efc8814a700 2024-03-29T10:57:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.688+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7efc80003ec0 con 0x7efc8814a700 2024-03-29T10:57:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.689+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7efc80084d10 con 0x7efc8814a700 2024-03-29T10:57:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.689+0000 7efc76ffd640 10 client.4423.objecter ms_dispatch 0x7efc88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.689+0000 7efc76ffd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.689+0000 7efc76ffd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:20.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.689+0000 7efc76ffd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T10:57:20.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.692+0000 7efc8659c640 10 monclient: get_auth_request con 0x7efc6806dac0 auth_method 0 2024-03-29T10:57:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.693+0000 7efc76ffd640 10 client.4423.objecter ms_handle_connect 0x7efc6806dac0 2024-03-29T10:57:20.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.696+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc800505f0 con 0x7efc8814a700 2024-03-29T10:57:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.696+0000 7efc76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:20.696+0000 7efc76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.026+0000 7efc8ce7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.026+0000 7efc8ce7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.026+0000 7efc8ce7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.026+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc6c005330 con 0x7efc8814a700 2024-03-29T10:57:21.011 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:21.012 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:21.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.031+0000 7efc76ffd640 1 -- 172.21.15.45:0/1316884 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7efc80052520 con 0x7efc8814a700 2024-03-29T10:57:21.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.031+0000 7efc76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.031+0000 7efc76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:21.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.035+0000 7efc8ce7d640 10 librados: watch_flush enter 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.035+0000 7efc8ce7d640 10 librados: watch_flush exit 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.035+0000 7efc8ce7d640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.035+0000 7efc8ce7d640 10 client.4423.objecter successfully canceled tick 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.035+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 >> v1:172.21.15.116:6816/3290579615 conn(0x7efc6806dac0 legacy=0x7efc6806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.035+0000 7efc8ce7d640 10 monclient: shutdown 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 >> v1:172.21.15.116:6789/0 conn(0x7efc8814a700 legacy=0x7efc881e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 shutdown_connections 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 >> 172.21.15.45:0/1316884 conn(0x7efc8806b660 msgr2=0x7efc880b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 shutdown_connections 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 1 -- 172.21.15.45:0/1316884 wait complete. 2024-03-29T10:57:21.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:21.036+0000 7efc8ce7d640 1 librados: shutdown 2024-03-29T10:57:21.037 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:27.843 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:27.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.013+0000 7ff362e25640 10 monclient: get_monmap_and_config 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.013+0000 7ff362e25640 10 monclient: build_initial_monmap 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.013+0000 7ff362e25640 10 monclient: monmap: 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:28.015282+0000 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:28.015282+0000 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:27.996 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:27.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.014+0000 7ff362e25640 1 Processor -- start 2024-03-29T10:57:27.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.015+0000 7ff362e25640 1 -- start start 2024-03-29T10:57:27.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.015+0000 7ff362e25640 10 monclient: init 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.015+0000 7ff362e25640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.015+0000 7ff362e25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.015+0000 7ff362e25640 10 monclient(hunting): picked mon.noname-b con 0x7ff35c0ae250 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.016+0000 7ff362e25640 10 monclient(hunting): picked mon.noname-c con 0x7ff35c13f970 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.016+0000 7ff362e25640 10 monclient(hunting): picked mon.noname-a con 0x7ff35c142800 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.018+0000 7ff360bc1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff35c0ae250 0x7ff35c13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33659/0 (socket says 172.21.15.45:33659) 2024-03-29T10:57:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.018+0000 7ff35bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff35c13f970 0x7ff35c1420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:46556/0 (socket says 172.21.15.45:46556) 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.018+0000 7ff360bc1640 1 -- 172.21.15.45:0/3687537707 learned_addr learned my addr 172.21.15.45:0/3687537707 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.018+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0afe00 con 0x7ff35c142800 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.018+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c142c10 con 0x7ff35c13f970 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0afb20 con 0x7ff35c0ae250 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff362e25640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff362e25640 10 monclient(hunting): authenticate will time out at 1122.289131s 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540993164 0 0) 0x7ff35c0afb20 con 0x7ff35c0ae250 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff340003520 con 0x7ff35c0ae250 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144380523 0 0) 0x7ff35c142c10 con 0x7ff35c13f970 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff35c0afb20 con 0x7ff35c13f970 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733606440 0 0) 0x7ff340003520 con 0x7ff35c0ae250 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 >> v1:172.21.15.116:6789/0 conn(0x7ff35c13f970 legacy=0x7ff35c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 >> v1:172.21.15.45:6789/0 conn(0x7ff35c142800 legacy=0x7ff35c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.019+0000 7ff35b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0aab80 con 0x7ff35c0ae250 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff348001690 con 0x7ff35c0ae250 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: dump: 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff362e25640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff362e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:28.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3480018b0 con 0x7ff35c0ae250 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 1 -- 172.21.15.45:0/3687537707 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff348001c40 con 0x7ff35c0ae250 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff35b7fe640 10 monclient: dump: 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff362e25640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:28.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.020+0000 7ff362e25640 4 set_mon_vals no callback set 2024-03-29T10:57:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.022+0000 7ff362e25640 10 monclient: shutdown 2024-03-29T10:57:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.022+0000 7ff362e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.022+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 >> v1:172.21.15.45:6790/0 conn(0x7ff35c0ae250 legacy=0x7ff35c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.022+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 shutdown_connections 2024-03-29T10:57:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.022+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 >> 172.21.15.45:0/3687537707 conn(0x7ff35c0b35e0 msgr2=0x7ff35c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.023+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 shutdown_connections 2024-03-29T10:57:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.023+0000 7ff362e25640 1 -- 172.21.15.45:0/3687537707 wait complete. 2024-03-29T10:57:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.024+0000 7ff362e25640 10 monclient: build_initial_monmap 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.024+0000 7ff362e25640 10 monclient: monmap: 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 1 librados: starting msgr at 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 1 librados: starting objecter 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 1 Processor -- start 2024-03-29T10:57:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 1 -- start start 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 1 librados: setting wanted keys 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 1 librados: calling monclient init 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.025+0000 7ff362e25640 10 monclient: init 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient(hunting): picked mon.noname-c con 0x7ff35c0ae250 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient(hunting): picked mon.noname-b con 0x7ff35c13f970 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient(hunting): picked mon.noname-a con 0x7ff35c142800 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c13be60 con 0x7ff35c142800 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c13c0f0 con 0x7ff35c13f970 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0acaa0 con 0x7ff35c0ae250 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:28.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.026+0000 7ff362e25640 10 monclient(hunting): authenticate will time out at 1122.296735s 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff35bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff35c13f970 0x7ff35c135d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:46868/0 (socket says 172.21.15.45:46868) 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff35bfff640 1 -- 172.21.15.45:0/1868287587 learned_addr learned my addr 172.21.15.45:0/1868287587 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 client.?.objecter ms_handle_connect 0x7ff35c0ae250 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 client.?.objecter ms_handle_connect 0x7ff35c142800 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 client.?.objecter ms_handle_connect 0x7ff35c13f970 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089778630 0 0) 0x7ff35c0acaa0 con 0x7ff35c0ae250 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff344004eb0 con 0x7ff35c0ae250 2024-03-29T10:57:28.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732984588 0 0) 0x7ff35c13be60 con 0x7ff35c142800 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440051d0 con 0x7ff35c142800 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969801335 0 0) 0x7ff35c13c0f0 con 0x7ff35c13f970 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.030+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff35c13be60 con 0x7ff35c13f970 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808687065 0 0) 0x7ff3440051d0 con 0x7ff35c142800 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff35c13c0f0 con 0x7ff35c142800 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039761233 0 0) 0x7ff344004eb0 con 0x7ff35c0ae250 2024-03-29T10:57:28.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3440051d0 con 0x7ff35c0ae250 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff34c001810 con 0x7ff35c142800 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): dump: 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff348001610 con 0x7ff35c0ae250 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient(hunting): dump: 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1219560336 0 0) 0x7ff35c13c0f0 con 0x7ff35c142800 2024-03-29T10:57:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 >> v1:172.21.15.45:6790/0 conn(0x7ff35c0ae250 legacy=0x7ff35c13b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 >> v1:172.21.15.116:6789/0 conn(0x7ff35c13f970 legacy=0x7ff35c135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 monclient: found mon.a 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0acc30 con 0x7ff35c142800 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff3597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 10 monclient: _renew_subs 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c136430 con 0x7ff35c142800 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 10 monclient: _renew_subs 2024-03-29T10:57:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:28.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c13b370 con 0x7ff35c142800 2024-03-29T10:57:28.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.031+0000 7ff362e25640 1 librados: init done 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.033+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff34c001d20 con 0x7ff35c142800 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.033+0000 7ff3597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.033+0000 7ff35affd640 4 set_mon_vals no callback set 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff34c0037b0 con 0x7ff35c142800 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff3597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff3597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff3597fa640 10 monclient: dump: 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.021 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7ff34c019780 con 0x7ff35c142800 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff352ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff352ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff352ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.034+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff338005080 con 0x7ff35c142800 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.035+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7ff34c083740 con 0x7ff35c142800 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.035+0000 7ff3597fa640 10 client.4404.objecter ms_dispatch 0x7ff35c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.035+0000 7ff3597fa640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.035+0000 7ff3597fa640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.035+0000 7ff3597fa640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T10:57:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.035+0000 7ff360bc1640 10 monclient: get_auth_request con 0x7ff34406e0e0 auth_method 0 2024-03-29T10:57:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.036+0000 7ff3597fa640 10 client.4404.objecter ms_handle_connect 0x7ff34406e0e0 2024-03-29T10:57:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.044+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff34c04f0e0 con 0x7ff35c142800 2024-03-29T10:57:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.045+0000 7ff3597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.045+0000 7ff3597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff352ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff352ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff352ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff338005330 con 0x7ff35c142800 2024-03-29T10:57:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff3597fa640 1 -- 172.21.15.45:0/1868287587 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7ff34c051010 con 0x7ff35c142800 2024-03-29T10:57:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff3597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.380+0000 7ff3597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:28.363 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:28.363 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:28.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.384+0000 7ff352ffd640 10 librados: watch_flush enter 2024-03-29T10:57:28.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.384+0000 7ff352ffd640 10 librados: watch_flush exit 2024-03-29T10:57:28.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.384+0000 7ff352ffd640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T10:57:28.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.384+0000 7ff352ffd640 10 client.4404.objecter successfully canceled tick 2024-03-29T10:57:28.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.384+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 >> v1:172.21.15.116:6816/3290579615 conn(0x7ff34406e0e0 legacy=0x7ff3440705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 10 monclient: shutdown 2024-03-29T10:57:28.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:28.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 >> v1:172.21.15.45:6789/0 conn(0x7ff35c142800 legacy=0x7ff35c1d7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 shutdown_connections 2024-03-29T10:57:28.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 >> 172.21.15.45:0/1868287587 conn(0x7ff35c0b35e0 msgr2=0x7ff35c0b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:28.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 shutdown_connections 2024-03-29T10:57:28.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.385+0000 7ff352ffd640 1 -- 172.21.15.45:0/1868287587 wait complete. 2024-03-29T10:57:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:28.388+0000 7ff352ffd640 1 librados: shutdown 2024-03-29T10:57:28.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:35.194 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:35.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.371+0000 7faa4ff90640 10 monclient: get_monmap_and_config 2024-03-29T10:57:35.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.371+0000 7faa4ff90640 10 monclient: build_initial_monmap 2024-03-29T10:57:35.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.371+0000 7faa4ff90640 10 monclient: monmap: 2024-03-29T10:57:35.353 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:35.373113+0000 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:35.373113+0000 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:35.354 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.372+0000 7faa4ff90640 1 Processor -- start 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.372+0000 7faa4ff90640 1 -- start start 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.372+0000 7faa4ff90640 10 monclient: init 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient(hunting): picked mon.noname-b con 0x7faa480b4100 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient(hunting): picked mon.noname-c con 0x7faa4813ee90 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient(hunting): picked mon.noname-a con 0x7faa480af900 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:35.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480afd10 con 0x7faa480af900 2024-03-29T10:57:35.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480af6f0 con 0x7faa4813ee90 2024-03-29T10:57:35.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480aac80 con 0x7faa480b4100 2024-03-29T10:57:35.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:35.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.373+0000 7faa4ff90640 10 monclient(hunting): authenticate will time out at 1129.643873s 2024-03-29T10:57:35.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4d52b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faa4813ee90 0x7faa4813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:51840/0 (socket says 172.21.15.45:51840) 2024-03-29T10:57:35.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4d52b640 1 -- 172.21.15.45:0/2776932075 learned_addr learned my addr 172.21.15.45:0/2776932075 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089853058 0 0) 0x7faa480af6f0 con 0x7faa4813ee90 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa34003520 con 0x7faa4813ee90 2024-03-29T10:57:35.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034813477 0 0) 0x7faa480afd10 con 0x7faa480af900 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480af6f0 con 0x7faa480af900 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479066320 0 0) 0x7faa34003520 con 0x7faa4813ee90 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 >> v1:172.21.15.45:6790/0 conn(0x7faa480b4100 legacy=0x7faa480adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 >> v1:172.21.15.45:6789/0 conn(0x7faa480af900 legacy=0x7faa4814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 monclient: found mon.noname-c 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.374+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480aaf60 con 0x7faa4813ee90 2024-03-29T10:57:35.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 10 monclient: _finish_auth 0 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faa44001690 con 0x7faa4813ee90 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4cd2a640 10 monclient: dump: 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:35.359 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4ff90640 5 monclient: authenticate success, global_id 4432 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.375+0000 7faa4ff90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa440017f0 con 0x7faa4813ee90 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4cd2a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4cd2a640 1 -- 172.21.15.45:0/2776932075 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faa44001c00 con 0x7faa4813ee90 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4cd2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4cd2a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4cd2a640 10 monclient: dump: 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.360 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4ff90640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:35.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4ff90640 4 set_mon_vals no callback set 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4ff90640 10 monclient: shutdown 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4ff90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.376+0000 7faa4ff90640 1 -- 172.21.15.45:0/2776932075 >> v1:172.21.15.116:6789/0 conn(0x7faa4813ee90 legacy=0x7faa4813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.377+0000 7faa4ff90640 1 -- 172.21.15.45:0/2776932075 shutdown_connections 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.377+0000 7faa4ff90640 1 -- 172.21.15.45:0/2776932075 >> 172.21.15.45:0/2776932075 conn(0x7faa480b35e0 msgr2=0x7faa480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.377+0000 7faa4ff90640 1 -- 172.21.15.45:0/2776932075 shutdown_connections 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.377+0000 7faa4ff90640 1 -- 172.21.15.45:0/2776932075 wait complete. 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.378+0000 7faa4ff90640 10 monclient: build_initial_monmap 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.378+0000 7faa4ff90640 10 monclient: monmap: 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:35.364 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.378+0000 7faa4ff90640 1 librados: starting msgr at 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.379+0000 7faa4ff90640 1 librados: starting objecter 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.379+0000 7faa4ff90640 1 Processor -- start 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.379+0000 7faa4ff90640 1 -- start start 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.379+0000 7faa4ff90640 1 librados: setting wanted keys 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.379+0000 7faa4ff90640 1 librados: calling monclient init 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.379+0000 7faa4ff90640 10 monclient: init 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:35.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient(hunting): picked mon.noname-a con 0x7faa480af900 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient(hunting): picked mon.noname-b con 0x7faa480b4100 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient(hunting): picked mon.noname-c con 0x7faa4813ee90 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4812b0d0 con 0x7faa480af900 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4812b3b0 con 0x7faa480b4100 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa48128ce0 con 0x7faa4813ee90 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4ff90640 10 monclient(hunting): authenticate will time out at 1129.650371s 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4dd2c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faa480af900 0x7faa4813c6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52684/0 (socket says 172.21.15.45:52684) 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa4dd2c640 1 -- 172.21.15.45:0/2623529002 learned_addr learned my addr 172.21.15.45:0/2623529002 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa3e7fc640 10 client.?.objecter ms_handle_connect 0x7faa480af900 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa3e7fc640 10 client.?.objecter ms_handle_connect 0x7faa480b4100 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.380+0000 7faa3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 client.?.objecter ms_handle_connect 0x7faa4813ee90 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932813596 0 0) 0x7faa4812b3b0 con 0x7faa480b4100 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa2c004eb0 con 0x7faa480b4100 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122304709 0 0) 0x7faa4812b0d0 con 0x7faa480af900 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:57:35.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4812b3b0 con 0x7faa480af900 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3245481266 0 0) 0x7faa2c004eb0 con 0x7faa480b4100 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa4812b0d0 con 0x7faa480b4100 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faa44001bc0 con 0x7faa480b4100 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:35.368 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.381+0000 7faa3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3068187323 0 0) 0x7faa4812b0d0 con 0x7faa480b4100 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 >> v1:172.21.15.45:6790/0 conn(0x7faa4813ee90 legacy=0x7faa481e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 >> v1:172.21.15.45:6789/0 conn(0x7faa480af900 legacy=0x7faa4813c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 1 monclient: found mon.b 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa48129e50 con 0x7faa480b4100 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 5 monclient: authenticate success, global_id 4435 2024-03-29T10:57:35.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 10 monclient: _renew_subs 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa481280e0 con 0x7faa480b4100 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 10 monclient: _renew_subs 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa48125920 con 0x7faa480b4100 2024-03-29T10:57:35.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.382+0000 7faa4ff90640 1 librados: init done 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa44001d40 con 0x7faa480b4100 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7faa44004c30 con 0x7faa480b4100 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3ffff640 4 set_mon_vals no callback set 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.383+0000 7faa3e7fc640 10 monclient: dump: 2024-03-29T10:57:35.371 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.384+0000 7faa4ff90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.384+0000 7faa4ff90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.384+0000 7faa4ff90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.384+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa30005080 con 0x7faa480b4100 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.385+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7faa4401ac00 con 0x7faa480b4100 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.386+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7faa44084a80 con 0x7faa480b4100 2024-03-29T10:57:35.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.386+0000 7faa3e7fc640 10 client.4435.objecter ms_dispatch 0x7faa48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:35.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.386+0000 7faa3e7fc640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:35.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.386+0000 7faa3e7fc640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:35.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.386+0000 7faa3e7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T10:57:35.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.386+0000 7faa4dd2c640 10 monclient: get_auth_request con 0x7faa2c06ddf0 auth_method 0 2024-03-29T10:57:35.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.387+0000 7faa3e7fc640 10 client.4435.objecter ms_handle_connect 0x7faa2c06ddf0 2024-03-29T10:57:35.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.393+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa44050360 con 0x7faa480b4100 2024-03-29T10:57:35.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.393+0000 7faa3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.393+0000 7faa3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:35.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.796+0000 7faa4ff90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:35.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.796+0000 7faa4ff90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:35.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.796+0000 7faa4ff90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:57:35.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.796+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa30005330 con 0x7faa480b4100 2024-03-29T10:57:35.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.798+0000 7faa3e7fc640 1 -- 172.21.15.45:0/2623529002 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7faa44052290 con 0x7faa480b4100 2024-03-29T10:57:35.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.798+0000 7faa3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:35.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.798+0000 7faa3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:35.780 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:35.780 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:35.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 10 librados: watch_flush enter 2024-03-29T10:57:35.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 10 librados: watch_flush exit 2024-03-29T10:57:35.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T10:57:35.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 10 client.4435.objecter successfully canceled tick 2024-03-29T10:57:35.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 >> v1:172.21.15.116:6816/3290579615 conn(0x7faa2c06ddf0 legacy=0x7faa2c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 10 monclient: shutdown 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.802+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 >> v1:172.21.15.116:6789/0 conn(0x7faa480b4100 legacy=0x7faa481df6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.803+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 shutdown_connections 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.803+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 >> 172.21.15.45:0/2623529002 conn(0x7faa480b35e0 msgr2=0x7faa48131370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.803+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 shutdown_connections 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.803+0000 7faa4ff90640 1 -- 172.21.15.45:0/2623529002 wait complete. 2024-03-29T10:57:35.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:35.803+0000 7faa4ff90640 1 librados: shutdown 2024-03-29T10:57:35.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:42.605 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:42.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.765+0000 7fcbe49f4640 10 monclient: get_monmap_and_config 2024-03-29T10:57:42.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.765+0000 7fcbe49f4640 10 monclient: build_initial_monmap 2024-03-29T10:57:42.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.765+0000 7fcbe49f4640 10 monclient: monmap: 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:42.766013+0000 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:42.766013+0000 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:42.746 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:42.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.766+0000 7fcbe49f4640 1 Processor -- start 2024-03-29T10:57:42.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.766+0000 7fcbe49f4640 1 -- start start 2024-03-29T10:57:42.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.766+0000 7fcbe49f4640 10 monclient: init 2024-03-29T10:57:42.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.767+0000 7fcbe49f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:42.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.767+0000 7fcbe49f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.767+0000 7fcbe49f4640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe00b0860 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.767+0000 7fcbe49f4640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe0072450 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.767+0000 7fcbe49f4640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe013ee40 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.767+0000 7fcbe49f4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00ad840 con 0x7fcbe013ee40 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbe49f4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe013c830 con 0x7fcbe00b0860 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbe49f4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe013c9c0 con 0x7fcbe0072450 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbe49f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbe49f4640 10 monclient(hunting): authenticate will time out at 1137.037129s 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbde59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcbe00b0860 0x7fcbe00acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:44944/0 (socket says 172.21.15.45:44944) 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbde59c640 1 -- 172.21.15.45:0/2312395374 learned_addr learned my addr 172.21.15.45:0/2312395374 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:42.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392024037 0 0) 0x7fcbe00ad840 con 0x7fcbe013ee40 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc8003520 con 0x7fcbe013ee40 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082620226 0 0) 0x7fcbe013c830 con 0x7fcbe00b0860 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00ad840 con 0x7fcbe00b0860 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334362088 0 0) 0x7fcbc8003520 con 0x7fcbe013ee40 2024-03-29T10:57:42.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.768+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 >> v1:172.21.15.45:6790/0 conn(0x7fcbe0072450 legacy=0x7fcbe013c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 >> v1:172.21.15.116:6789/0 conn(0x7fcbe00b0860 legacy=0x7fcbe00acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 1 monclient: found mon.noname-a 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe013f970 con 0x7fcbe013ee40 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: _finish_auth 0 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fcbd8001a10 con 0x7fcbe013ee40 2024-03-29T10:57:42.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: dump: 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbe49f4640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbe49f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd8001b70 con 0x7fcbe013ee40 2024-03-29T10:57:42.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 1 -- 172.21.15.45:0/2312395374 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fcbd8001fc0 con 0x7fcbe013ee40 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbdd59a640 10 monclient: dump: 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:42.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbe49f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:42.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.772+0000 7fcbe49f4640 4 set_mon_vals no callback set 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.773+0000 7fcbe49f4640 10 monclient: shutdown 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.773+0000 7fcbe49f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.773+0000 7fcbe49f4640 1 -- 172.21.15.45:0/2312395374 >> v1:172.21.15.45:6789/0 conn(0x7fcbe013ee40 legacy=0x7fcbe013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.773+0000 7fcbe49f4640 1 -- 172.21.15.45:0/2312395374 shutdown_connections 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.773+0000 7fcbe49f4640 1 -- 172.21.15.45:0/2312395374 >> 172.21.15.45:0/2312395374 conn(0x7fcbe006b660 msgr2=0x7fcbe006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.774+0000 7fcbe49f4640 1 -- 172.21.15.45:0/2312395374 shutdown_connections 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.774+0000 7fcbe49f4640 1 -- 172.21.15.45:0/2312395374 wait complete. 2024-03-29T10:57:42.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.775+0000 7fcbe49f4640 10 monclient: build_initial_monmap 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.775+0000 7fcbe49f4640 10 monclient: monmap: 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.776+0000 7fcbe49f4640 1 librados: starting msgr at 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.776+0000 7fcbe49f4640 1 librados: starting objecter 2024-03-29T10:57:42.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.776+0000 7fcbe49f4640 1 Processor -- start 2024-03-29T10:57:42.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.777+0000 7fcbe49f4640 1 -- start start 2024-03-29T10:57:42.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.777+0000 7fcbe49f4640 1 librados: setting wanted keys 2024-03-29T10:57:42.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.777+0000 7fcbe49f4640 1 librados: calling monclient init 2024-03-29T10:57:42.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.777+0000 7fcbe49f4640 10 monclient: init 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe0072450 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe00b0860 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe013ee40 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe0070460 con 0x7fcbe0072450 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00716d0 con 0x7fcbe013ee40 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe01316b0 con 0x7fcbe00b0860 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbe49f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbe49f4640 10 monclient(hunting): authenticate will time out at 1137.048000s 2024-03-29T10:57:42.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.778+0000 7fcbde59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcbe0072450 0x7fcbe00708f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56476/0 (socket says 172.21.15.45:56476) 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbde59c640 1 -- 172.21.15.45:0/648359287 learned_addr learned my addr 172.21.15.45:0/648359287 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 client.?.objecter ms_handle_connect 0x7fcbe0072450 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251655693 0 0) 0x7fcbe0070460 con 0x7fcbe0072450 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0004eb0 con 0x7fcbe0072450 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3657313262 0 0) 0x7fcbc0004eb0 con 0x7fcbe0072450 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbe0070460 con 0x7fcbe0072450 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fcbd0001a50 con 0x7fcbe0072450 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): dump: 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.779+0000 7fcbceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.780+0000 7fcbceffd640 10 client.?.objecter ms_handle_connect 0x7fcbe013ee40 2024-03-29T10:57:42.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.780+0000 7fcbceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.780+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 992299681 0 0) 0x7fcbe0070460 con 0x7fcbe0072450 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.780+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 >> v1:172.21.15.45:6790/0 conn(0x7fcbe00b0860 legacy=0x7fcbe00b39d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 >> v1:172.21.15.116:6789/0 conn(0x7fcbe013ee40 legacy=0x7fcbe0131db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 1 monclient: found mon.a 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0131990 con 0x7fcbe0072450 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbe49f4640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd0001d70 con 0x7fcbe0072450 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbe49f4640 10 monclient: _renew_subs 2024-03-29T10:57:42.767 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbe49f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.782+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe006f910 con 0x7fcbe0072450 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbe49f4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbe49f4640 10 monclient: _renew_subs 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbe49f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe00a94e0 con 0x7fcbe0072450 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbe49f4640 1 librados: init done 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.783+0000 7fcbdcd99640 4 set_mon_vals no callback set 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.784+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fcbd0004c30 con 0x7fcbe0072450 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.784+0000 7fcbceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.784+0000 7fcbceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.784+0000 7fcbceffd640 10 monclient: dump: 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:42.768 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:42.769 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.785+0000 7fcbe49f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.785+0000 7fcbe49f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.785+0000 7fcbe49f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.785+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe00b0c40 con 0x7fcbe0072450 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.787+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fcbd001ac00 con 0x7fcbe0072450 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.787+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fcbd0084c30 con 0x7fcbe0072450 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.787+0000 7fcbceffd640 10 client.4416.objecter ms_dispatch 0x7fcbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:42.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.787+0000 7fcbceffd640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:42.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.787+0000 7fcbceffd640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:42.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.788+0000 7fcbceffd640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T10:57:42.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.788+0000 7fcbddd9b640 10 monclient: get_auth_request con 0x7fcbc006db80 auth_method 0 2024-03-29T10:57:42.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.791+0000 7fcbceffd640 10 client.4416.objecter ms_handle_connect 0x7fcbc006db80 2024-03-29T10:57:42.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.795+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbd0050340 con 0x7fcbe0072450 2024-03-29T10:57:42.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.795+0000 7fcbceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:42.776 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:42.795+0000 7fcbceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:43.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.119+0000 7fcbe49f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.120+0000 7fcbe49f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.120+0000 7fcbe49f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:57:43.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.120+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbe0075f10 con 0x7fcbe0072450 2024-03-29T10:57:43.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.121+0000 7fcbceffd640 1 -- 172.21.15.45:0/648359287 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fcbd0052440 con 0x7fcbe0072450 2024-03-29T10:57:43.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.121+0000 7fcbceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.121+0000 7fcbceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:43.106 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:43.106 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.125+0000 7fcbe49f4640 10 librados: watch_flush enter 2024-03-29T10:57:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 10 librados: watch_flush exit 2024-03-29T10:57:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T10:57:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 10 client.4416.objecter successfully canceled tick 2024-03-29T10:57:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 >> v1:172.21.15.116:6816/3290579615 conn(0x7fcbc006db80 legacy=0x7fcbc0070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 10 monclient: shutdown 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 >> v1:172.21.15.45:6789/0 conn(0x7fcbe0072450 legacy=0x7fcbe00708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 shutdown_connections 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 >> 172.21.15.45:0/648359287 conn(0x7fcbe006b660 msgr2=0x7fcbe00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.126+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 shutdown_connections 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.127+0000 7fcbe49f4640 1 -- 172.21.15.45:0/648359287 wait complete. 2024-03-29T10:57:43.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:43.127+0000 7fcbe49f4640 1 librados: shutdown 2024-03-29T10:57:43.124 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:49.931 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.109+0000 7fb09c427640 10 monclient: get_monmap_and_config 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.109+0000 7fb09c427640 10 monclient: build_initial_monmap 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.109+0000 7fb09c427640 10 monclient: monmap: 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:50.110619+0000 2024-03-29T10:57:50.090 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:50.110619+0000 2024-03-29T10:57:50.091 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:50.091 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:50.091 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:50.091 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:50.091 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:50.091 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:50.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.110+0000 7fb09c427640 1 Processor -- start 2024-03-29T10:57:50.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.110+0000 7fb09c427640 1 -- start start 2024-03-29T10:57:50.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.110+0000 7fb09c427640 10 monclient: init 2024-03-29T10:57:50.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.111+0000 7fb09c427640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:50.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.111+0000 7fb09c427640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:50.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.111+0000 7fb09c427640 10 monclient(hunting): picked mon.noname-c con 0x7fb0940b4330 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 10 monclient(hunting): picked mon.noname-b con 0x7fb0941470b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 10 monclient(hunting): picked mon.noname-a con 0x7fb0941439b0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09406f2a0 con 0x7fb0941439b0 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb094146b90 con 0x7fb0940b4330 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940b2ca0 con 0x7fb0941470b0 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:50.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.112+0000 7fb09c427640 10 monclient(hunting): authenticate will time out at 1144.381388s 2024-03-29T10:57:50.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.115+0000 7fb09a9c4640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb0941439b0 0x7fb094151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51043/0 (socket says 172.21.15.45:51043) 2024-03-29T10:57:50.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.115+0000 7fb09a9c4640 1 -- 172.21.15.45:0/2202600855 learned_addr learned my addr 172.21.15.45:0/2202600855 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:50.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325965344 0 0) 0x7fb09406f2a0 con 0x7fb0941439b0 2024-03-29T10:57:50.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:57:50.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb080003520 con 0x7fb0941439b0 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496708431 0 0) 0x7fb094146b90 con 0x7fb0940b4330 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb09406f2a0 con 0x7fb0940b4330 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 532017586 0 0) 0x7fb080003520 con 0x7fb0941439b0 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 >> v1:172.21.15.45:6790/0 conn(0x7fb0941470b0 legacy=0x7fb0940b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 >> v1:172.21.15.116:6789/0 conn(0x7fb0940b4330 legacy=0x7fb0940b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: _finish_hunting 0 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 monclient: found mon.noname-a 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:57:50.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09406e0c0 con 0x7fb0941439b0 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: _finish_auth 0 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: _check_auth_tickets 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb084001690 con 0x7fb0941439b0 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.117+0000 7fb0991c1640 10 monclient: dump: 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:50.106 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.118+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0840017f0 con 0x7fb0941439b0 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.118+0000 7fb0991c1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.118+0000 7fb0991c1640 1 -- 172.21.15.45:0/2202600855 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb084001c00 con 0x7fb0941439b0 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.118+0000 7fb0991c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.118+0000 7fb0991c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.118+0000 7fb0991c1640 10 monclient: dump: 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.120+0000 7fb09c427640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.120+0000 7fb09c427640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:50.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.120+0000 7fb09c427640 4 set_mon_vals no callback set 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.121+0000 7fb09c427640 10 monclient: shutdown 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.121+0000 7fb09c427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.121+0000 7fb09c427640 1 -- 172.21.15.45:0/2202600855 >> v1:172.21.15.45:6789/0 conn(0x7fb0941439b0 legacy=0x7fb094151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.121+0000 7fb09c427640 1 -- 172.21.15.45:0/2202600855 shutdown_connections 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.121+0000 7fb09c427640 1 -- 172.21.15.45:0/2202600855 >> 172.21.15.45:0/2202600855 conn(0x7fb094066a00 msgr2=0x7fb094066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.122+0000 7fb09c427640 1 -- 172.21.15.45:0/2202600855 shutdown_connections 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.122+0000 7fb09c427640 1 -- 172.21.15.45:0/2202600855 wait complete. 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.123+0000 7fb09c427640 10 monclient: build_initial_monmap 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.123+0000 7fb09c427640 10 monclient: monmap: 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:50.109 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:50.110 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:50.110 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:50.110 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:50.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.123+0000 7fb09c427640 1 librados: starting msgr at 2024-03-29T10:57:50.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.123+0000 7fb09c427640 1 librados: starting objecter 2024-03-29T10:57:50.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.124+0000 7fb09c427640 1 Processor -- start 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.124+0000 7fb09c427640 1 -- start start 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.124+0000 7fb09c427640 1 librados: setting wanted keys 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.124+0000 7fb09c427640 1 librados: calling monclient init 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.124+0000 7fb09c427640 10 monclient: init 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient(hunting): picked mon.noname-c con 0x7fb0940b4330 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient(hunting): picked mon.noname-a con 0x7fb0941439b0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient(hunting): picked mon.noname-b con 0x7fb0941261b0 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb094127140 con 0x7fb0941439b0 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0941db530 con 0x7fb0941261b0 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb094122680 con 0x7fb0940b4330 2024-03-29T10:57:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.125+0000 7fb09c427640 10 monclient(hunting): authenticate will time out at 1144.394308s 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.128+0000 7fb09a1c3640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb0940b4330 0x7fb0941dba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56207/0 (socket says 172.21.15.45:56207) 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.128+0000 7fb09a1c3640 1 -- 172.21.15.45:0/425459510 learned_addr learned my addr 172.21.15.45:0/425459510 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.128+0000 7fb08affd640 10 client.?.objecter ms_handle_connect 0x7fb0940b4330 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680320040 0 0) 0x7fb094122680 con 0x7fb0940b4330 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb078004eb0 con 0x7fb0940b4330 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 client.?.objecter ms_handle_connect 0x7fb0941439b0 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.129+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506562959 0 0) 0x7fb078004eb0 con 0x7fb0940b4330 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb094122680 con 0x7fb0940b4330 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb090001a90 con 0x7fb0940b4330 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 10 monclient(hunting): dump: 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2874356911 0 0) 0x7fb094122680 con 0x7fb0940b4330 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.130+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 >> v1:172.21.15.116:6789/0 conn(0x7fb0941261b0 legacy=0x7fb0941e6e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 >> v1:172.21.15.45:6789/0 conn(0x7fb0941439b0 legacy=0x7fb094140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 1 monclient: found mon.c 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0941dc160 con 0x7fb0940b4330 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 10 monclient: _finish_auth 0 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb08affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 10 monclient: _renew_subs 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0941dc410 con 0x7fb0940b4330 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 10 monclient: _renew_subs 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094141680 con 0x7fb0940b4330 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.131+0000 7fb09c427640 1 librados: init done 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.132+0000 7fb09c427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.133+0000 7fb09c427640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.133+0000 7fb09c427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.133+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094075f10 con 0x7fb0940b4330 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.138+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb090001bf0 con 0x7fb0940b4330 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.138+0000 7fb08affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.138+0000 7fb08bfff640 4 set_mon_vals no callback set 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.139+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb090003b20 con 0x7fb0940b4330 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.139+0000 7fb08affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.139+0000 7fb08affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.139+0000 7fb08affd640 10 monclient: dump: 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:50.121 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.139+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fb090003d40 con 0x7fb0940b4330 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb0999c2640 10 monclient: get_auth_request con 0x7fb07806dc30 auth_method 0 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fb090084c60 con 0x7fb0940b4330 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb08affd640 10 client.4439.objecter ms_dispatch 0x7fb094000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb08affd640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb08affd640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb08affd640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T10:57:50.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.140+0000 7fb08affd640 10 client.4439.objecter ms_handle_connect 0x7fb07806dc30 2024-03-29T10:57:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.149+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0900505c0 con 0x7fb0940b4330 2024-03-29T10:57:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.149+0000 7fb08affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.149+0000 7fb08affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:50.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.486+0000 7fb09c427640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.486+0000 7fb09c427640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.486+0000 7fb09c427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.486+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0940b74e0 con 0x7fb0940b4330 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.487+0000 7fb08affd640 1 -- 172.21.15.45:0/425459510 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fb0900524f0 con 0x7fb0940b4330 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.487+0000 7fb08affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.487+0000 7fb08affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:50.469 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:50.469 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 10 librados: watch_flush enter 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 10 librados: watch_flush exit 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 10 client.4439.objecter successfully canceled tick 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 >> v1:172.21.15.116:6816/3290579615 conn(0x7fb07806dc30 legacy=0x7fb0780700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 10 monclient: shutdown 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.492+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 >> v1:172.21.15.45:6790/0 conn(0x7fb0940b4330 legacy=0x7fb0941dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.493+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 shutdown_connections 2024-03-29T10:57:50.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.493+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 >> 172.21.15.45:0/425459510 conn(0x7fb094066a00 msgr2=0x7fb094070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:50.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.496+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 shutdown_connections 2024-03-29T10:57:50.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.499+0000 7fb09c427640 1 -- 172.21.15.45:0/425459510 wait complete. 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:50.503+0000 7fb09c427640 1 librados: shutdown 2024-03-29T10:57:50.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:57.309 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:57.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.478+0000 7fb0fdb2b640 10 monclient: get_monmap_and_config 2024-03-29T10:57:57.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.478+0000 7fb0fdb2b640 10 monclient: build_initial_monmap 2024-03-29T10:57:57.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.478+0000 7fb0fdb2b640 10 monclient: monmap: 2024-03-29T10:57:57.459 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:57.459 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:57.459 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:57:57.479720+0000 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:57:57.479720+0000 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.480+0000 7fb0fdb2b640 1 Processor -- start 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.480+0000 7fb0fdb2b640 1 -- start start 2024-03-29T10:57:57.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.480+0000 7fb0fdb2b640 10 monclient: init 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0f81422b0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f8142690 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0f80b17d0 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f8142c80 con 0x7fb0f81422b0 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80aae40 con 0x7fb0f8142690 2024-03-29T10:57:57.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f813fbd0 con 0x7fb0f80b17d0 2024-03-29T10:57:57.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:57.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0fdb2b640 10 monclient(hunting): authenticate will time out at 1151.750118s 2024-03-29T10:57:57.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0f7fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb0f80b17d0 0x7fb0f80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46630/0 (socket says 172.21.15.45:46630) 2024-03-29T10:57:57.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.481+0000 7fb0f7fff640 1 -- 172.21.15.45:0/3593744528 learned_addr learned my addr 172.21.15.45:0/3593744528 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:57.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811629282 0 0) 0x7fb0f813fbd0 con 0x7fb0f80b17d0 2024-03-29T10:57:57.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0003520 con 0x7fb0f80b17d0 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666379587 0 0) 0x7fb0e0003520 con 0x7fb0f80b17d0 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 >> v1:172.21.15.116:6789/0 conn(0x7fb0f8142690 legacy=0x7fb0f80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:57.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.482+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 >> v1:172.21.15.45:6789/0 conn(0x7fb0f81422b0 legacy=0x7fb0f813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:57.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:57.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 1 monclient: found mon.noname-b 2024-03-29T10:57:57.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:57.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:57:57.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f8151940 con 0x7fb0f80b17d0 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0fdb2b640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0fdb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0ec001690 con 0x7fb0f80b17d0 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: dump: 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.465 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:57.466 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:57.466 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:57.466 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:57.466 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:57.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0fdb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:57.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 1 -- 172.21.15.45:0/3593744528 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0ec0017f0 con 0x7fb0f80b17d0 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0fdb2b640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.484+0000 7fb0fdb2b640 4 set_mon_vals no callback set 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.485+0000 7fb0fdb2b640 10 monclient: shutdown 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.485+0000 7fb0fdb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.485+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3593744528 >> v1:172.21.15.45:6790/0 conn(0x7fb0f80b17d0 legacy=0x7fb0f80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.485+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3593744528 shutdown_connections 2024-03-29T10:57:57.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.485+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3593744528 >> 172.21.15.45:0/3593744528 conn(0x7fb0f8066a00 msgr2=0x7fb0f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.488+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3593744528 shutdown_connections 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.488+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3593744528 wait complete. 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.489+0000 7fb0fdb2b640 10 monclient: build_initial_monmap 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.489+0000 7fb0fdb2b640 10 monclient: monmap: 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:57.471 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.489+0000 7fb0fdb2b640 1 librados: starting msgr at 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.489+0000 7fb0fdb2b640 1 librados: starting objecter 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.489+0000 7fb0fdb2b640 1 Processor -- start 2024-03-29T10:57:57.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 1 -- start start 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 1 librados: setting wanted keys 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 1 librados: calling monclient init 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient: init 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0f80b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0f81422b0 addr v1:172.21.15.45:6789/0 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f8142690 addr v1:172.21.15.45:6790/0 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80ab490 con 0x7fb0f81422b0 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80ac440 con 0x7fb0f80b17d0 2024-03-29T10:57:57.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80ac6b0 con 0x7fb0f8142690 2024-03-29T10:57:57.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:57.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.490+0000 7fb0fdb2b640 10 monclient(hunting): authenticate will time out at 1151.759907s 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0f7fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb0f8142690 0x7fb0f8086070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55224/0 (socket says 172.21.15.45:55224) 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0f7fff640 1 -- 172.21.15.45:0/3649136592 learned_addr learned my addr 172.21.15.45:0/3649136592 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 10 client.?.objecter ms_handle_connect 0x7fb0f8142690 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803966421 0 0) 0x7fb0f80ac6b0 con 0x7fb0f8142690 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:57:57.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.494+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e4004eb0 con 0x7fb0f8142690 2024-03-29T10:57:57.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 client.?.objecter ms_handle_connect 0x7fb0f81422b0 2024-03-29T10:57:57.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:57.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4100215041 0 0) 0x7fb0e4004eb0 con 0x7fb0f8142690 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e4005300 con 0x7fb0f8142690 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914666558 0 0) 0x7fb0f80ab490 con 0x7fb0f81422b0 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e4004eb0 con 0x7fb0f81422b0 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0ec004ae0 con 0x7fb0f8142690 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): dump: 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.477 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1806800683 0 0) 0x7fb0e4004eb0 con 0x7fb0f81422b0 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f80ab490 con 0x7fb0f81422b0 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0e80016a0 con 0x7fb0f81422b0 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.495+0000 7fb0dffff640 10 monclient(hunting): dump: 2024-03-29T10:57:57.478 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 283814838 0 0) 0x7fb0e4005300 con 0x7fb0f8142690 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 >> v1:172.21.15.116:6789/0 conn(0x7fb0f80b17d0 legacy=0x7fb0f80b6e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 >> v1:172.21.15.45:6789/0 conn(0x7fb0f81422b0 legacy=0x7fb0f812de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 1 monclient: found mon.c 2024-03-29T10:57:57.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:57.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:57.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f80ac990 con 0x7fb0f8142690 2024-03-29T10:57:57.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 10 monclient: _finish_auth 0 2024-03-29T10:57:57.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:57.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:57.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0ec004ef0 con 0x7fb0f8142690 2024-03-29T10:57:57.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:57.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.496+0000 7fb0f5ffb640 4 set_mon_vals no callback set 2024-03-29T10:57:57.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.497+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb0ec005340 con 0x7fb0f8142690 2024-03-29T10:57:57.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.497+0000 7fb0dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.497+0000 7fb0dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.497+0000 7fb0dffff640 10 monclient: dump: 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:57:57.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 10 monclient: _renew_subs 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f80ab6e0 con 0x7fb0f8142690 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 10 monclient: _renew_subs 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f81325a0 con 0x7fb0f8142690 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.498+0000 7fb0fdb2b640 1 librados: init done 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.500+0000 7fb0fdb2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.500+0000 7fb0fdb2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.500+0000 7fb0fdb2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.500+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f8075f10 con 0x7fb0f8142690 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.501+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fb0ec001aa0 con 0x7fb0f8142690 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.501+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fb0ec054b00 con 0x7fb0f8142690 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.501+0000 7fb0dffff640 10 client.4445.objecter ms_dispatch 0x7fb0f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:57.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.501+0000 7fb0dffff640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:57.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.501+0000 7fb0dffff640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:57.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.501+0000 7fb0dffff640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T10:57:57.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.502+0000 7fb0f77fe640 10 monclient: get_auth_request con 0x7fb0e406e250 auth_method 0 2024-03-29T10:57:57.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.504+0000 7fb0dffff640 10 client.4445.objecter ms_handle_connect 0x7fb0e406e250 2024-03-29T10:57:57.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.512+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0ec050da0 con 0x7fb0f8142690 2024-03-29T10:57:57.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.512+0000 7fb0dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.512+0000 7fb0dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:57.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.843+0000 7fb0fdb2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:57.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.843+0000 7fb0fdb2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:57.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.843+0000 7fb0fdb2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:57:57.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.843+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0f80b2270 con 0x7fb0f8142690 2024-03-29T10:57:57.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.844+0000 7fb0dffff640 1 -- 172.21.15.45:0/3649136592 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fb0ec050f40 con 0x7fb0f8142690 2024-03-29T10:57:57.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.844+0000 7fb0dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:57.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.844+0000 7fb0dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:57.825 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:57:57.826 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.848+0000 7fb0fdb2b640 10 librados: watch_flush enter 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 10 librados: watch_flush exit 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 10 client.4445.objecter successfully canceled tick 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 >> v1:172.21.15.116:6816/3290579615 conn(0x7fb0e406e250 legacy=0x7fb0e4070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 10 monclient: shutdown 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 >> v1:172.21.15.45:6790/0 conn(0x7fb0f8142690 legacy=0x7fb0f8086070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 shutdown_connections 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 >> 172.21.15.45:0/3649136592 conn(0x7fb0f8066a00 msgr2=0x7fb0f80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 shutdown_connections 2024-03-29T10:57:57.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 -- 172.21.15.45:0/3649136592 wait complete. 2024-03-29T10:57:57.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:57:57.849+0000 7fb0fdb2b640 1 librados: shutdown 2024-03-29T10:57:57.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:04.656 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.826+0000 7f957957b640 10 monclient: get_monmap_and_config 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.826+0000 7f957957b640 10 monclient: build_initial_monmap 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.826+0000 7f957957b640 10 monclient: monmap: 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:04.827639+0000 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:04.827639+0000 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:04.807 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:04.808 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:04.808 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:04.808 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:04.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.829+0000 7f957957b640 1 Processor -- start 2024-03-29T10:58:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.830+0000 7f957957b640 1 -- start start 2024-03-29T10:58:04.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.830+0000 7f957957b640 10 monclient: init 2024-03-29T10:58:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.830+0000 7f957957b640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.830+0000 7f957957b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.830+0000 7f957957b640 10 monclient(hunting): picked mon.noname-c con 0x7f95741457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:04.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.830+0000 7f957957b640 10 monclient(hunting): picked mon.noname-a con 0x7f957413f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f957957b640 10 monclient(hunting): picked mon.noname-b con 0x7f95740a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f957957b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95740aaef0 con 0x7f957413f650 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f957957b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574145bd0 con 0x7f95741457f0 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f957957b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574145eb0 con 0x7f95740a9d70 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f957957b640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f957957b640 10 monclient(hunting): authenticate will time out at 1159.100740s 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f95737fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f95740a9d70 0x7f957414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38570/0 (socket says 172.21.15.45:38570) 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.831+0000 7f95737fe640 1 -- 172.21.15.45:0/8120108 learned_addr learned my addr 172.21.15.45:0/8120108 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732205546 0 0) 0x7f9574145eb0 con 0x7f95740a9d70 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:04.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:58:04.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c00bd30 con 0x7f95740a9d70 2024-03-29T10:58:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085576569 0 0) 0x7f955c00bd30 con 0x7f95740a9d70 2024-03-29T10:58:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.832+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 >> v1:172.21.15.116:6789/0 conn(0x7f95741457f0 legacy=0x7f957413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.833+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 >> v1:172.21.15.45:6789/0 conn(0x7f957413f650 legacy=0x7f95740b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 1 monclient: found mon.noname-b 2024-03-29T10:58:04.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:58:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574146340 con 0x7f95740a9d70 2024-03-29T10:58:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:04.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f956c001a10 con 0x7f95740a9d70 2024-03-29T10:58:04.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: dump: 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f957957b640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:58:04.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f957957b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f956c001be0 con 0x7f95740a9d70 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 1 -- 172.21.15.45:0/8120108 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f956c002030 con 0x7f95740a9d70 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f9571ffb640 10 monclient: dump: 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:04.821 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f957957b640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.834+0000 7f957957b640 4 set_mon_vals no callback set 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.835+0000 7f957957b640 10 monclient: shutdown 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.835+0000 7f957957b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.835+0000 7f957957b640 1 -- 172.21.15.45:0/8120108 >> v1:172.21.15.45:6790/0 conn(0x7f95740a9d70 legacy=0x7f957414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.835+0000 7f957957b640 1 -- 172.21.15.45:0/8120108 shutdown_connections 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.835+0000 7f957957b640 1 -- 172.21.15.45:0/8120108 >> 172.21.15.45:0/8120108 conn(0x7f9574066a00 msgr2=0x7f9574066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.840+0000 7f957957b640 1 -- 172.21.15.45:0/8120108 shutdown_connections 2024-03-29T10:58:04.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.840+0000 7f957957b640 1 -- 172.21.15.45:0/8120108 wait complete. 2024-03-29T10:58:04.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.841+0000 7f957957b640 10 monclient: build_initial_monmap 2024-03-29T10:58:04.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.841+0000 7f957957b640 10 monclient: monmap: 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 1 librados: starting msgr at 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 1 librados: starting objecter 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 1 Processor -- start 2024-03-29T10:58:04.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 1 -- start start 2024-03-29T10:58:04.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 1 librados: setting wanted keys 2024-03-29T10:58:04.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 1 librados: calling monclient init 2024-03-29T10:58:04.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.842+0000 7f957957b640 10 monclient: init 2024-03-29T10:58:04.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.843+0000 7f957957b640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:04.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.843+0000 7f957957b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.843+0000 7f957957b640 10 monclient(hunting): picked mon.noname-b con 0x7f95740a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.843+0000 7f957957b640 10 monclient(hunting): picked mon.noname-a con 0x7f957413f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.844+0000 7f957957b640 10 monclient(hunting): picked mon.noname-c con 0x7f95741457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.844+0000 7f957957b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574144d00 con 0x7f957413f650 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.844+0000 7f957957b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574135da0 con 0x7f95740a9d70 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.844+0000 7f957957b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574145120 con 0x7f95741457f0 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.844+0000 7f957957b640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:04.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.844+0000 7f957957b640 10 monclient(hunting): authenticate will time out at 1159.113053s 2024-03-29T10:58:04.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.847+0000 7f9572ffd640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f95740a9d70 legacy=0x7f95741436a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.116:6789/0 2024-03-29T10:58:04.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.847+0000 7f95737fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f95741457f0 0x7f95741dda10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43124/0 (socket says 172.21.15.45:43124) 2024-03-29T10:58:04.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.847+0000 7f95737fe640 1 -- 172.21.15.45:0/2717737720 learned_addr learned my addr 172.21.15.45:0/2717737720 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 client.?.objecter ms_handle_connect 0x7f95741457f0 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425094511 0 0) 0x7f9574145120 con 0x7f95741457f0 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9560004eb0 con 0x7f95741457f0 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 client.?.objecter ms_handle_connect 0x7f957413f650 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4274497408 0 0) 0x7f9560004eb0 con 0x7f95741457f0 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9574145120 con 0x7f95741457f0 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f956c001ef0 con 0x7f95741457f0 2024-03-29T10:58:04.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): dump: 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:04.830 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:04.831 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:04.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.848+0000 7f955b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:04.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1009471925 0 0) 0x7f9574145120 con 0x7f95741457f0 2024-03-29T10:58:04.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 >> v1:172.21.15.116:6789/0 conn(0x7f95740a9d70 legacy=0x7f95741436a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 >> v1:172.21.15.45:6789/0 conn(0x7f957413f650 legacy=0x7f957413c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 1 monclient: found mon.c 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95741452b0 con 0x7f95741457f0 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.849+0000 7f955b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:58:04.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 10 monclient: _renew_subs 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957413dfa0 con 0x7f95741457f0 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 10 monclient: _renew_subs 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9574134f30 con 0x7f95741457f0 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f957957b640 1 librados: init done 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f956c004f70 con 0x7f95741457f0 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f955b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f956c005280 con 0x7f95741457f0 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f955b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f955b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f955b7fe640 10 monclient: dump: 2024-03-29T10:58:04.835 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.850+0000 7f95717fa640 4 set_mon_vals no callback set 2024-03-29T10:58:04.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.853+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f956c01b250 con 0x7f95741457f0 2024-03-29T10:58:04.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f956c084f90 con 0x7f95741457f0 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f955b7fe640 10 client.4451.objecter ms_dispatch 0x7f9574000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f955b7fe640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f955b7fe640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f955b7fe640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f9572ffd640 10 monclient: get_auth_request con 0x7f956006dc10 auth_method 0 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.854+0000 7f957957b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.855+0000 7f955b7fe640 10 client.4451.objecter ms_handle_connect 0x7f956006dc10 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.855+0000 7f957957b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.855+0000 7f957957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:04.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.855+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9554005080 con 0x7f95741457f0 2024-03-29T10:58:04.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.863+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f956c0b28b0 con 0x7f95741457f0 2024-03-29T10:58:04.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.863+0000 7f955b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:04.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:04.863+0000 7f955b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:05.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.189+0000 7f957957b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.189+0000 7f957957b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.190+0000 7f957957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:05.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.190+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9554005330 con 0x7f95741457f0 2024-03-29T10:58:05.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.191+0000 7f955b7fe640 1 -- 172.21.15.45:0/2717737720 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f956c0506a0 con 0x7f95741457f0 2024-03-29T10:58:05.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.191+0000 7f955b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.191+0000 7f955b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:05.171 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:05.171 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 10 librados: watch_flush enter 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 10 librados: watch_flush exit 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 10 client.4451.objecter successfully canceled tick 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 >> v1:172.21.15.116:6816/3290579615 conn(0x7f956006dc10 legacy=0x7f95600700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 10 monclient: shutdown 2024-03-29T10:58:05.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:05.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 >> v1:172.21.15.45:6790/0 conn(0x7f95741457f0 legacy=0x7f95741dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 shutdown_connections 2024-03-29T10:58:05.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 >> 172.21.15.45:0/2717737720 conn(0x7f9574066a00 msgr2=0x7f957413a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:05.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 shutdown_connections 2024-03-29T10:58:05.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 -- 172.21.15.45:0/2717737720 wait complete. 2024-03-29T10:58:05.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:05.194+0000 7f957957b640 1 librados: shutdown 2024-03-29T10:58:05.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:11.999 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:12.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.170+0000 7fb058e72640 10 monclient: get_monmap_and_config 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.170+0000 7fb058e72640 10 monclient: build_initial_monmap 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.170+0000 7fb058e72640 10 monclient: monmap: 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:12.171776+0000 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:12.171776+0000 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:12.152 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:12.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.172+0000 7fb058e72640 1 Processor -- start 2024-03-29T10:58:12.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.172+0000 7fb058e72640 1 -- start start 2024-03-29T10:58:12.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.172+0000 7fb058e72640 10 monclient: init 2024-03-29T10:58:12.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:12.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:12.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient(hunting): picked mon.noname-b con 0x7fb0540ac310 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient(hunting): picked mon.noname-c con 0x7fb0540ae460 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient(hunting): picked mon.noname-a con 0x7fb05413d270 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0540add30 con 0x7fb05413d270 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05406df00 con 0x7fb0540ae460 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0540ac6f0 con 0x7fb0540ac310 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.173+0000 7fb058e72640 10 monclient(hunting): authenticate will time out at 1166.442120s 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.174+0000 7fb052ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb0540ae460 0x7fb05413cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:45080/0 (socket says 172.21.15.45:45080) 2024-03-29T10:58:12.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.174+0000 7fb052ffd640 1 -- 172.21.15.45:0/947726416 learned_addr learned my addr 172.21.15.45:0/947726416 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:12.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524107321 0 0) 0x7fb05406df00 con 0x7fb0540ae460 2024-03-29T10:58:12.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c003520 con 0x7fb0540ae460 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450768608 0 0) 0x7fb0540ac6f0 con 0x7fb0540ac310 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.176+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05406df00 con 0x7fb0540ac310 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.177+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190213131 0 0) 0x7fb05406df00 con 0x7fb0540ac310 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.177+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 >> v1:172.21.15.116:6789/0 conn(0x7fb0540ae460 legacy=0x7fb05413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.177+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 >> v1:172.21.15.45:6789/0 conn(0x7fb05413d270 legacy=0x7fb05414b4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:12.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 1 monclient: found mon.noname-b 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05406dc20 con 0x7fb0540ac310 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb04c0018e0 con 0x7fb0540ac310 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: dump: 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:12.162 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb04c001ab0 con 0x7fb0540ac310 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 1 -- 172.21.15.45:0/947726416 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb04c001e40 con 0x7fb0540ac310 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.179+0000 7fb0527fc640 10 monclient: dump: 2024-03-29T10:58:12.163 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.180+0000 7fb058e72640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.180+0000 7fb058e72640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:12.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.180+0000 7fb058e72640 4 set_mon_vals no callback set 2024-03-29T10:58:12.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.180+0000 7fb058e72640 10 monclient: shutdown 2024-03-29T10:58:12.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.180+0000 7fb058e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:12.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.180+0000 7fb058e72640 1 -- 172.21.15.45:0/947726416 >> v1:172.21.15.45:6790/0 conn(0x7fb0540ac310 legacy=0x7fb0541403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.181+0000 7fb058e72640 1 -- 172.21.15.45:0/947726416 shutdown_connections 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.181+0000 7fb058e72640 1 -- 172.21.15.45:0/947726416 >> 172.21.15.45:0/947726416 conn(0x7fb0540b3a30 msgr2=0x7fb0540b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.181+0000 7fb058e72640 1 -- 172.21.15.45:0/947726416 shutdown_connections 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.181+0000 7fb058e72640 1 -- 172.21.15.45:0/947726416 wait complete. 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.182+0000 7fb058e72640 10 monclient: build_initial_monmap 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.182+0000 7fb058e72640 10 monclient: monmap: 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:12.167 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.182+0000 7fb058e72640 1 librados: starting msgr at 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.182+0000 7fb058e72640 1 librados: starting objecter 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.182+0000 7fb058e72640 1 Processor -- start 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.183+0000 7fb058e72640 1 -- start start 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.183+0000 7fb058e72640 1 librados: setting wanted keys 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.183+0000 7fb058e72640 1 librados: calling monclient init 2024-03-29T10:58:12.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.183+0000 7fb058e72640 10 monclient: init 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient(hunting): picked mon.noname-a con 0x7fb0540ac310 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient(hunting): picked mon.noname-c con 0x7fb0540ae460 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient(hunting): picked mon.noname-b con 0x7fb05413d270 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0541dc110 con 0x7fb0540ac310 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0540aaf20 con 0x7fb05413d270 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0541db200 con 0x7fb0540ae460 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:12.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.184+0000 7fb058e72640 10 monclient(hunting): authenticate will time out at 1166.453456s 2024-03-29T10:58:12.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.187+0000 7fb052ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fb0540ae460 0x7fb0540aa810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48381/0 (socket says 172.21.15.45:48381) 2024-03-29T10:58:12.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.187+0000 7fb052ffd640 1 -- 172.21.15.45:0/817744940 learned_addr learned my addr 172.21.15.45:0/817744940 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:12.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.188+0000 7fb03bfff640 10 client.?.objecter ms_handle_connect 0x7fb0540ac310 2024-03-29T10:58:12.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.188+0000 7fb03bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.189+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872180339 0 0) 0x7fb0541dc110 con 0x7fb0540ac310 2024-03-29T10:58:12.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.189+0000 7fb03bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.189+0000 7fb03bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.189+0000 7fb03bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.189+0000 7fb03bfff640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:58:12.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.189+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb040004eb0 con 0x7fb0540ac310 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 client.?.objecter ms_handle_connect 0x7fb0540ae460 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3554708817 0 0) 0x7fb040004eb0 con 0x7fb0540ac310 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb040005300 con 0x7fb0540ac310 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb04c001e40 con 0x7fb0540ac310 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): dump: 2024-03-29T10:58:12.176 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:12.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 client.?.objecter ms_handle_connect 0x7fb05413d270 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443903336 0 0) 0x7fb0541db200 con 0x7fb0540ae460 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.191+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb040004eb0 con 0x7fb0540ae460 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.192+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026401689 0 0) 0x7fb0540aaf20 con 0x7fb05413d270 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.192+0000 7fb03bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.192+0000 7fb03bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.192+0000 7fb03bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.192+0000 7fb03bfff640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.192+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0541db200 con 0x7fb05413d270 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1532456427 0 0) 0x7fb040005300 con 0x7fb0540ac310 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 >> v1:172.21.15.45:6790/0 conn(0x7fb0540ae460 legacy=0x7fb0540aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 >> v1:172.21.15.116:6789/0 conn(0x7fb05413d270 legacy=0x7fb0541339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:12.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 monclient: found mon.a 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0541db4e0 con 0x7fb0540ac310 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: _finish_auth 0 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:12.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb04c003c20 con 0x7fb0540ac310 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb04c003f30 con 0x7fb0540ac310 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb03bfff640 10 monclient: dump: 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:12.181 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb058e72640 5 monclient: authenticate success, global_id 4428 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.193+0000 7fb051ffb640 4 set_mon_vals no callback set 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 10 monclient: _renew_subs 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0540a9d30 con 0x7fb0540ac310 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 10 monclient: _renew_subs 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0540ab3a0 con 0x7fb0540ac310 2024-03-29T10:58:12.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.194+0000 7fb058e72640 1 librados: init done 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.197+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fb04c019f00 con 0x7fb0540ac310 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.197+0000 7fb058e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.197+0000 7fb058e72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.197+0000 7fb058e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.197+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0540ac960 con 0x7fb0540ac310 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb052ffd640 10 monclient: get_auth_request con 0x7fb04006df50 auth_method 0 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fb04c053940 con 0x7fb0540ac310 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb03bfff640 10 client.4428.objecter ms_dispatch 0x7fb054000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb03bfff640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb03bfff640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb03bfff640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T10:58:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.198+0000 7fb03bfff640 10 client.4428.objecter ms_handle_connect 0x7fb04006df50 2024-03-29T10:58:12.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.207+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb04c04fbe0 con 0x7fb0540ac310 2024-03-29T10:58:12.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.207+0000 7fb03bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.207+0000 7fb03bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:12.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.527+0000 7fb058e72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.527+0000 7fb058e72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.527+0000 7fb058e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:12.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.527+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0541db670 con 0x7fb0540ac310 2024-03-29T10:58:12.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.528+0000 7fb03bfff640 1 -- 172.21.15.45:0/817744940 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fb04c04fd80 con 0x7fb0540ac310 2024-03-29T10:58:12.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.528+0000 7fb03bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.528+0000 7fb03bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:12.510 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:12.511 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:12.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 10 librados: watch_flush enter 2024-03-29T10:58:12.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 10 librados: watch_flush exit 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 10 client.4428.objecter successfully canceled tick 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 >> v1:172.21.15.116:6816/3290579615 conn(0x7fb04006df50 legacy=0x7fb040070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 10 monclient: shutdown 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.531+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 >> v1:172.21.15.45:6789/0 conn(0x7fb0540ac310 legacy=0x7fb0541dba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.532+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 shutdown_connections 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.532+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 >> 172.21.15.45:0/817744940 conn(0x7fb0540b3a30 msgr2=0x7fb054136ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.533+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 shutdown_connections 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.533+0000 7fb058e72640 1 -- 172.21.15.45:0/817744940 wait complete. 2024-03-29T10:58:12.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:12.533+0000 7fb058e72640 1 librados: shutdown 2024-03-29T10:58:12.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:19.337 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:19.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.509+0000 7f0219b32640 10 monclient: get_monmap_and_config 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.509+0000 7f0219b32640 10 monclient: build_initial_monmap 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.509+0000 7f0219b32640 10 monclient: monmap: 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:19.510708+0000 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:19.510708+0000 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:19.491 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:19.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.511+0000 7f0219b32640 1 Processor -- start 2024-03-29T10:58:19.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.511+0000 7f0219b32640 1 -- start start 2024-03-29T10:58:19.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.511+0000 7f0219b32640 10 monclient: init 2024-03-29T10:58:19.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:19.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient(hunting): picked mon.noname-a con 0x7f02140b0770 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient(hunting): picked mon.noname-c con 0x7f021414a700 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient(hunting): picked mon.noname-b con 0x7f02140ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140aba70 con 0x7f02140b0770 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140aecb0 con 0x7f021414a700 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140af150 con 0x7f02140ac5a0 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0219b32640 10 monclient(hunting): authenticate will time out at 1173.781393s 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0212ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f021414a700 0x7f02140abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:54212/0 (socket says 172.21.15.45:54212) 2024-03-29T10:58:19.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.512+0000 7f0212ffd640 1 -- 172.21.15.45:0/3568841540 learned_addr learned my addr 172.21.15.45:0/3568841540 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:19.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741232636 0 0) 0x7f02140aecb0 con 0x7f021414a700 2024-03-29T10:58:19.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01fc003520 con 0x7f021414a700 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027161299 0 0) 0x7f02140af150 con 0x7f02140ac5a0 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.513+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02140aecb0 con 0x7f02140ac5a0 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632621440 0 0) 0x7f01fc003520 con 0x7f021414a700 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 >> v1:172.21.15.45:6790/0 conn(0x7f02140ac5a0 legacy=0x7f0214151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 >> v1:172.21.15.45:6789/0 conn(0x7f02140b0770 legacy=0x7f0214149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 1 monclient: found mon.noname-c 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140ac9d0 con 0x7f021414a700 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f02080016b0 con 0x7f021414a700 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f02127fc640 10 monclient: dump: 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:19.499 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f0219b32640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.514+0000 7f0219b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0208001810 con 0x7f021414a700 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f02127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f02127fc640 1 -- 172.21.15.45:0/3568841540 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0208001c20 con 0x7f021414a700 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f02127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f02127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:19.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f02127fc640 10 monclient: dump: 2024-03-29T10:58:19.501 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:19.501 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:19.501 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f0219b32640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.515+0000 7f0219b32640 4 set_mon_vals no callback set 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 10 monclient: shutdown 2024-03-29T10:58:19.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:19.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 1 -- 172.21.15.45:0/3568841540 >> v1:172.21.15.116:6789/0 conn(0x7f021414a700 legacy=0x7f02140abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 1 -- 172.21.15.45:0/3568841540 shutdown_connections 2024-03-29T10:58:19.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 1 -- 172.21.15.45:0/3568841540 >> 172.21.15.45:0/3568841540 conn(0x7f021406b660 msgr2=0x7f021406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:19.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 1 -- 172.21.15.45:0/3568841540 shutdown_connections 2024-03-29T10:58:19.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.516+0000 7f0219b32640 1 -- 172.21.15.45:0/3568841540 wait complete. 2024-03-29T10:58:19.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.517+0000 7f0219b32640 10 monclient: build_initial_monmap 2024-03-29T10:58:19.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.517+0000 7f0219b32640 10 monclient: monmap: 2024-03-29T10:58:19.512 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:19.512 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:19.512 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:19.512 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 1 librados: starting msgr at 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 1 librados: starting objecter 2024-03-29T10:58:19.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 1 Processor -- start 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 1 -- start start 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 1 librados: setting wanted keys 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 1 librados: calling monclient init 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.518+0000 7f0219b32640 10 monclient: init 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient(hunting): picked mon.noname-b con 0x7f02140ac5a0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient(hunting): picked mon.noname-a con 0x7f02140b0770 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient(hunting): picked mon.noname-c con 0x7f021414a700 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:19.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214126d60 con 0x7f02140b0770 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214127ea0 con 0x7f02140ac5a0 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214128180 con 0x7f021414a700 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.519+0000 7f0219b32640 10 monclient(hunting): authenticate will time out at 1173.788763s 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.520+0000 7f02137fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f02140ac5a0 0x7f0214144d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:54840/0 (socket says 172.21.15.45:54840) 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.520+0000 7f02137fe640 1 -- 172.21.15.45:0/345930913 learned_addr learned my addr 172.21.15.45:0/345930913 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:19.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.520+0000 7f01fbfff640 10 client.?.objecter ms_handle_connect 0x7f02140ac5a0 2024-03-29T10:58:19.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.520+0000 7f01fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.520+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470902071 0 0) 0x7f0214127ea0 con 0x7f02140ac5a0 2024-03-29T10:58:19.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:58:19.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200004eb0 con 0x7f02140ac5a0 2024-03-29T10:58:19.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 client.?.objecter ms_handle_connect 0x7f021414a700 2024-03-29T10:58:19.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 client.?.objecter ms_handle_connect 0x7f02140b0770 2024-03-29T10:58:19.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866559972 0 0) 0x7f0214128180 con 0x7f021414a700 2024-03-29T10:58:19.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:58:19.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0214127ea0 con 0x7f021414a700 2024-03-29T10:58:19.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.521+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1564567798 0 0) 0x7f0200004eb0 con 0x7f02140ac5a0 2024-03-29T10:58:19.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0214128180 con 0x7f02140ac5a0 2024-03-29T10:58:19.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011772471 0 0) 0x7f0214126d60 con 0x7f02140b0770 2024-03-29T10:58:19.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:58:19.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200004eb0 con 0x7f02140b0770 2024-03-29T10:58:19.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0204001a30 con 0x7f02140ac5a0 2024-03-29T10:58:19.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:19.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): dump: 2024-03-29T10:58:19.521 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:19.521 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:19.521 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.521 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.521 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.522 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:19.522 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:19.522 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:19.522 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:19.522 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:19.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:19.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.522+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3627353461 0 0) 0x7f0214128180 con 0x7f02140ac5a0 2024-03-29T10:58:19.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 >> v1:172.21.15.45:6790/0 conn(0x7f021414a700 legacy=0x7f02141e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 >> v1:172.21.15.45:6789/0 conn(0x7f02140b0770 legacy=0x7f02141dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:19.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 1 monclient: found mon.b 2024-03-29T10:58:19.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:19.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:19.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214125e00 con 0x7f02140ac5a0 2024-03-29T10:58:19.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 10 monclient: _finish_auth 0 2024-03-29T10:58:19.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:19.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f01fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:19.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 5 monclient: authenticate success, global_id 4456 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 10 monclient: _renew_subs 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02141271f0 con 0x7f02140ac5a0 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 10 monclient: _renew_subs 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0214126500 con 0x7f02140ac5a0 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.523+0000 7f0219b32640 1 librados: init done 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0204001b90 con 0x7f02140ac5a0 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f01fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0204003a60 con 0x7f02140ac5a0 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f01fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f01fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f01fbfff640 10 monclient: dump: 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:19.526 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.524+0000 7f02117fa640 4 set_mon_vals no callback set 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.525+0000 7f0219b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.526+0000 7f0219b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.526+0000 7f0219b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.526+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f4005080 con 0x7f02140ac5a0 2024-03-29T10:58:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.526+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f0204003bc0 con 0x7f02140ac5a0 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.527+0000 7f0212ffd640 10 monclient: get_auth_request con 0x7f020006dc90 auth_method 0 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.527+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f0204084b70 con 0x7f02140ac5a0 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.527+0000 7f01fbfff640 10 client.4456.objecter ms_dispatch 0x7f0214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.527+0000 7f01fbfff640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.527+0000 7f01fbfff640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.527+0000 7f01fbfff640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T10:58:19.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.528+0000 7f01fbfff640 10 client.4456.objecter ms_handle_connect 0x7f020006dc90 2024-03-29T10:58:19.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.535+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0204050450 con 0x7f02140ac5a0 2024-03-29T10:58:19.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.535+0000 7f01fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.535+0000 7f01fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:19.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.956+0000 7f0219b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:19.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.956+0000 7f0219b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:19.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.956+0000 7f0219b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:19.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.956+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01f4005330 con 0x7f02140ac5a0 2024-03-29T10:58:19.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.957+0000 7f01fbfff640 1 -- 172.21.15.45:0/345930913 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f0204052380 con 0x7f02140ac5a0 2024-03-29T10:58:19.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.957+0000 7f01fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:19.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.957+0000 7f01fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:19.938 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:19.938 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 10 librados: watch_flush enter 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 10 librados: watch_flush exit 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 10 client.4456.objecter successfully canceled tick 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 >> v1:172.21.15.116:6816/3290579615 conn(0x7f020006dc90 legacy=0x7f0200070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 10 monclient: shutdown 2024-03-29T10:58:19.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:19.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 >> v1:172.21.15.116:6789/0 conn(0x7f02140ac5a0 legacy=0x7f0214144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 shutdown_connections 2024-03-29T10:58:19.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.961+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 >> 172.21.15.45:0/345930913 conn(0x7f021406b660 msgr2=0x7f0214147d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:19.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.962+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 shutdown_connections 2024-03-29T10:58:19.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.962+0000 7f0219b32640 1 -- 172.21.15.45:0/345930913 wait complete. 2024-03-29T10:58:19.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:19.962+0000 7f0219b32640 1 librados: shutdown 2024-03-29T10:58:19.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:26.766 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.931+0000 7f692de33640 10 monclient: get_monmap_and_config 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.931+0000 7f692de33640 10 monclient: build_initial_monmap 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.931+0000 7f692de33640 10 monclient: monmap: 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:26.932238+0000 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:26.932238+0000 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:26.914 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:26.915 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:26.915 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:26.915 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:26.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.933+0000 7f692de33640 1 Processor -- start 2024-03-29T10:58:26.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 1 -- start start 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient: init 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient(hunting): picked mon.noname-a con 0x7f6920034850 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient(hunting): picked mon.noname-b con 0x7f692002a640 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient(hunting): picked mon.noname-c con 0x7f69200327d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692002c820 con 0x7f6920034850 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692002a120 con 0x7f69200327d0 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200c75b0 con 0x7f692002a640 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:26.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.934+0000 7f692de33640 10 monclient(hunting): authenticate will time out at 1181.203903s 2024-03-29T10:58:26.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.937+0000 7f6927fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f692002a640 0x7f69200320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:52746/0 (socket says 172.21.15.45:52746) 2024-03-29T10:58:26.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.937+0000 7f6927fff640 1 -- 172.21.15.45:0/2533548207 learned_addr learned my addr 172.21.15.45:0/2533548207 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792500824 0 0) 0x7f69200c75b0 con 0x7f692002a640 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6918003520 con 0x7f692002a640 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449190358 0 0) 0x7f692002c820 con 0x7f6920034850 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200c75b0 con 0x7f6920034850 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690017041 0 0) 0x7f692002a120 con 0x7f69200327d0 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:58:26.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.938+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692002c820 con 0x7f69200327d0 2024-03-29T10:58:26.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433312444 0 0) 0x7f6918003520 con 0x7f692002a640 2024-03-29T10:58:26.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 >> v1:172.21.15.116:6789/0 conn(0x7f69200327d0 legacy=0x7f69200cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 >> v1:172.21.15.45:6789/0 conn(0x7f6920034850 legacy=0x7f69200c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:26.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 1 monclient: found mon.noname-b 2024-03-29T10:58:26.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920033110 con 0x7f692002a640 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6914001690 con 0x7f692002a640 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:26.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f69277fe640 10 monclient: dump: 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f692de33640 5 monclient: authenticate success, global_id 4466 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.939+0000 7f692de33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69140017f0 con 0x7f692002a640 2024-03-29T10:58:26.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f69277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f69277fe640 1 -- 172.21.15.45:0/2533548207 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6914001c40 con 0x7f692002a640 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f69277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f69277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f69277fe640 10 monclient: dump: 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:26.924 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:26.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f692de33640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:26.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.940+0000 7f692de33640 4 set_mon_vals no callback set 2024-03-29T10:58:26.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.941+0000 7f692de33640 10 monclient: shutdown 2024-03-29T10:58:26.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.941+0000 7f692de33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.941+0000 7f692de33640 1 -- 172.21.15.45:0/2533548207 >> v1:172.21.15.45:6790/0 conn(0x7f692002a640 legacy=0x7f69200320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.942+0000 7f692de33640 1 -- 172.21.15.45:0/2533548207 shutdown_connections 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.942+0000 7f692de33640 1 -- 172.21.15.45:0/2533548207 >> 172.21.15.45:0/2533548207 conn(0x7f69200058f0 msgr2=0x7f6920005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.942+0000 7f692de33640 1 -- 172.21.15.45:0/2533548207 shutdown_connections 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.942+0000 7f692de33640 1 -- 172.21.15.45:0/2533548207 wait complete. 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.943+0000 7f692de33640 10 monclient: build_initial_monmap 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.943+0000 7f692de33640 10 monclient: monmap: 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:26.927 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.943+0000 7f692de33640 1 librados: starting msgr at 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.943+0000 7f692de33640 1 librados: starting objecter 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.944+0000 7f692de33640 1 Processor -- start 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.944+0000 7f692de33640 1 -- start start 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.944+0000 7f692de33640 1 librados: setting wanted keys 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.944+0000 7f692de33640 1 librados: calling monclient init 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.944+0000 7f692de33640 10 monclient: init 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:26.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:26.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient(hunting): picked mon.noname-b con 0x7f692002a640 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:26.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient(hunting): picked mon.noname-c con 0x7f69200327d0 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:26.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient(hunting): picked mon.noname-a con 0x7f6920034850 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:26.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200c3e90 con 0x7f6920034850 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200c4170 con 0x7f692002a640 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200bc8f0 con 0x7f69200327d0 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.945+0000 7f692de33640 10 monclient(hunting): authenticate will time out at 1181.214702s 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.949+0000 7f692d632640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6920034850 0x7f69200bc1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45467/0 (socket says 172.21.15.45:45467) 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.949+0000 7f692d632640 1 -- 172.21.15.45:0/4074509259 learned_addr learned my addr 172.21.15.45:0/4074509259 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.949+0000 7f69257fa640 10 client.?.objecter ms_handle_connect 0x7f6920034850 2024-03-29T10:58:26.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.949+0000 7f69257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:26.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 client.?.objecter ms_handle_connect 0x7f69200327d0 2024-03-29T10:58:26.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374826611 0 0) 0x7f69200c3e90 con 0x7f6920034850 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c004eb0 con 0x7f6920034850 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156186955 0 0) 0x7f69200bc8f0 con 0x7f69200327d0 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.950+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200c3e90 con 0x7f69200327d0 2024-03-29T10:58:26.934 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 client.?.objecter ms_handle_connect 0x7f692002a640 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3777886514 0 0) 0x7f69200c3e90 con 0x7f69200327d0 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69200bc8f0 con 0x7f69200327d0 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6914001b00 con 0x7f69200327d0 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): dump: 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:26.935 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592585234 0 0) 0x7f69200c4170 con 0x7f692002a640 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200c3e90 con 0x7f692002a640 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3874178153 0 0) 0x7f69200bc8f0 con 0x7f69200327d0 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 >> v1:172.21.15.116:6789/0 conn(0x7f692002a640 legacy=0x7f69200c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 >> v1:172.21.15.45:6789/0 conn(0x7f6920034850 legacy=0x7f69200bc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 monclient: found mon.c 2024-03-29T10:58:26.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692002ddc0 con 0x7f69200327d0 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.951+0000 7f69257fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6914001ce0 con 0x7f69200327d0 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6914004e10 con 0x7f69200327d0 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f6926ffd640 4 set_mon_vals no callback set 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f69257fa640 10 monclient: dump: 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:26.937 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 10 monclient: _renew_subs 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6920031c90 con 0x7f69200327d0 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 10 monclient: _renew_subs 2024-03-29T10:58:26.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:26.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69200c1e30 con 0x7f69200327d0 2024-03-29T10:58:26.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.952+0000 7f692de33640 1 librados: init done 2024-03-29T10:58:26.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.955+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f691401ade0 con 0x7f69200327d0 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f692de33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f692de33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f692de33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69200c1500 con 0x7f69200327d0 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f692ce31640 10 monclient: get_auth_request con 0x7f690c06ddf0 auth_method 0 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f6914084b80 con 0x7f69200327d0 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f69257fa640 10 client.4469.objecter ms_dispatch 0x7f6928000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f69257fa640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f69257fa640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.956+0000 7f69257fa640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T10:58:26.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.959+0000 7f69257fa640 10 client.4469.objecter ms_handle_connect 0x7f690c06ddf0 2024-03-29T10:58:26.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.964+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6914050460 con 0x7f69200327d0 2024-03-29T10:58:26.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.964+0000 7f69257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:26.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:26.964+0000 7f69257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:27.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.300+0000 7f692de33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.300+0000 7f692de33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.300+0000 7f692de33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:27.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.300+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6920004cd0 con 0x7f69200327d0 2024-03-29T10:58:27.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.302+0000 7f69257fa640 1 -- 172.21.15.45:0/4074509259 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f6914052390 con 0x7f69200327d0 2024-03-29T10:58:27.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.302+0000 7f69257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.302+0000 7f69257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:27.284 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:27.284 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.306+0000 7f692de33640 10 librados: watch_flush enter 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.306+0000 7f692de33640 10 librados: watch_flush exit 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.306+0000 7f692de33640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.306+0000 7f692de33640 10 client.4469.objecter successfully canceled tick 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.307+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 >> v1:172.21.15.116:6816/3290579615 conn(0x7f690c06ddf0 legacy=0x7f690c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.307+0000 7f692de33640 10 monclient: shutdown 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.307+0000 7f692de33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:27.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.307+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 >> v1:172.21.15.45:6790/0 conn(0x7f69200327d0 legacy=0x7f69200bbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.307+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 shutdown_connections 2024-03-29T10:58:27.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.307+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 >> 172.21.15.45:0/4074509259 conn(0x7f69200058f0 msgr2=0x7f69200043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:27.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.309+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 shutdown_connections 2024-03-29T10:58:27.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.309+0000 7f692de33640 1 -- 172.21.15.45:0/4074509259 wait complete. 2024-03-29T10:58:27.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:27.310+0000 7f692de33640 1 librados: shutdown 2024-03-29T10:58:27.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:34.114 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.280+0000 7fc849be3640 10 monclient: get_monmap_and_config 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.280+0000 7fc849be3640 10 monclient: build_initial_monmap 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.281+0000 7fc849be3640 10 monclient: monmap: 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:34.282836+0000 2024-03-29T10:58:34.263 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:34.282836+0000 2024-03-29T10:58:34.264 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:34.264 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:34.264 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:34.264 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:34.264 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:34.264 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.282+0000 7fc849be3640 1 Processor -- start 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.282+0000 7fc849be3640 1 -- start start 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.282+0000 7fc849be3640 10 monclient: init 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.283+0000 7fc849be3640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.283+0000 7fc849be3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.283+0000 7fc849be3640 10 monclient(hunting): picked mon.noname-c con 0x7fc83c034850 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.283+0000 7fc849be3640 10 monclient(hunting): picked mon.noname-b con 0x7fc83c0c9550 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:34.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc848be1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc83c034850 0x7fc83c0c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:54870/0 (socket says 172.21.15.45:54870) 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc849be3640 10 monclient(hunting): picked mon.noname-a con 0x7fc83c02aac0 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc849be3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c02c440 con 0x7fc83c02aac0 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc849be3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c02aed0 con 0x7fc83c034850 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc849be3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c02b190 con 0x7fc83c0c9550 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc849be3640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc849be3640 10 monclient(hunting): authenticate will time out at 1188.554431s 2024-03-29T10:58:34.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.284+0000 7fc848be1640 1 -- 172.21.15.45:0/3598578843 learned_addr learned my addr 172.21.15.45:0/3598578843 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115756727 0 0) 0x7fc83c02c440 con 0x7fc83c02aac0 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc834003520 con 0x7fc83c02aac0 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465909782 0 0) 0x7fc83c02b190 con 0x7fc83c0c9550 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.287+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc83c02c440 con 0x7fc83c0c9550 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4001728547 0 0) 0x7fc834003520 con 0x7fc83c02aac0 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 >> v1:172.21.15.45:6790/0 conn(0x7fc83c0c9550 legacy=0x7fc83c0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 >> v1:172.21.15.116:6789/0 conn(0x7fc83c034850 legacy=0x7fc83c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 monclient: found mon.noname-a 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c033060 con 0x7fc83c02aac0 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:34.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc830001690 con 0x7fc83c02aac0 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: dump: 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:34.273 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc830001860 con 0x7fc83c02aac0 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 1 -- 172.21.15.45:0/3598578843 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc830001c70 con 0x7fc83c02aac0 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc8437fe640 10 monclient: dump: 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.274 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc849be3640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc849be3640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.288+0000 7fc849be3640 4 set_mon_vals no callback set 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.289+0000 7fc849be3640 10 monclient: shutdown 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.289+0000 7fc849be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.289+0000 7fc849be3640 1 -- 172.21.15.45:0/3598578843 >> v1:172.21.15.45:6789/0 conn(0x7fc83c02aac0 legacy=0x7fc83c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.290+0000 7fc849be3640 1 -- 172.21.15.45:0/3598578843 shutdown_connections 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.290+0000 7fc849be3640 1 -- 172.21.15.45:0/3598578843 >> 172.21.15.45:0/3598578843 conn(0x7fc83c0058f0 msgr2=0x7fc83c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.291+0000 7fc849be3640 1 -- 172.21.15.45:0/3598578843 shutdown_connections 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.291+0000 7fc849be3640 1 -- 172.21.15.45:0/3598578843 wait complete. 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 10 monclient: build_initial_monmap 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 10 monclient: monmap: 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:34.277 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 1 librados: starting msgr at 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 1 librados: starting objecter 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 1 Processor -- start 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 1 -- start start 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 1 librados: setting wanted keys 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 1 librados: calling monclient init 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.292+0000 7fc849be3640 10 monclient: init 2024-03-29T10:58:34.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient(hunting): picked mon.noname-c con 0x7fc83c02aac0 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient(hunting): picked mon.noname-a con 0x7fc83c034850 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient(hunting): picked mon.noname-b con 0x7fc83c0c9550 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c164150 con 0x7fc83c034850 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0bdaa0 con 0x7fc83c0c9550 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0bdc30 con 0x7fc83c02aac0 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.293+0000 7fc849be3640 10 monclient(hunting): authenticate will time out at 1188.563957s 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.294+0000 7fc8493e2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc83c0c9550 0x7fc83c0be190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:58278/0 (socket says 172.21.15.45:58278) 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.294+0000 7fc8493e2640 1 -- 172.21.15.45:0/42004222 learned_addr learned my addr 172.21.15.45:0/42004222 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.294+0000 7fc8417fa640 10 client.?.objecter ms_handle_connect 0x7fc83c0c9550 2024-03-29T10:58:34.279 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.294+0000 7fc8417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:34.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.295+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827581065 0 0) 0x7fc83c0bdaa0 con 0x7fc83c0c9550 2024-03-29T10:58:34.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.295+0000 7fc8417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.295+0000 7fc8417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.295+0000 7fc8417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.295+0000 7fc8417fa640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:58:34.280 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.295+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc828004eb0 con 0x7fc83c0c9550 2024-03-29T10:58:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3354638620 0 0) 0x7fc828004eb0 con 0x7fc83c0c9550 2024-03-29T10:58:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc83c0bdaa0 con 0x7fc83c0c9550 2024-03-29T10:58:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc830001b70 con 0x7fc83c0c9550 2024-03-29T10:58:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 10 monclient(hunting): dump: 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2985187302 0 0) 0x7fc83c0bdaa0 con 0x7fc83c0c9550 2024-03-29T10:58:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.296+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 >> v1:172.21.15.45:6790/0 conn(0x7fc83c02aac0 legacy=0x7fc83c161130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 >> v1:172.21.15.45:6789/0 conn(0x7fc83c034850 legacy=0x7fc83c164670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 1 monclient: found mon.b 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c161840 con 0x7fc83c0c9550 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 10 client.?.objecter ms_handle_connect 0x7fc83c034850 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc8417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:34.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc849be3640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc849be3640 10 monclient: _renew_subs 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc849be3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.300+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c161af0 con 0x7fc83c0c9550 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.301+0000 7fc849be3640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.301+0000 7fc849be3640 10 monclient: _renew_subs 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.301+0000 7fc849be3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.301+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c165070 con 0x7fc83c0c9550 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.301+0000 7fc849be3640 1 librados: init done 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc830003940 con 0x7fc83c0c9550 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc8417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc830003d10 con 0x7fc83c0c9550 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc8417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc8417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc8417fa640 10 monclient: dump: 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc842ffd640 4 set_mon_vals no callback set 2024-03-29T10:58:34.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.302+0000 7fc849be3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.303+0000 7fc849be3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.303+0000 7fc849be3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.303+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824005080 con 0x7fc83c0c9550 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.304+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fc830003e70 con 0x7fc83c0c9550 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.305+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fc830084af0 con 0x7fc83c0c9550 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.305+0000 7fc8417fa640 10 client.4468.objecter ms_dispatch 0x7fc844000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.305+0000 7fc8417fa640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.305+0000 7fc8417fa640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.305+0000 7fc8417fa640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T10:58:34.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.306+0000 7fc848be1640 10 monclient: get_auth_request con 0x7fc82806dab0 auth_method 0 2024-03-29T10:58:34.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.306+0000 7fc8417fa640 10 client.4468.objecter ms_handle_connect 0x7fc82806dab0 2024-03-29T10:58:34.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.311+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8300503d0 con 0x7fc83c0c9550 2024-03-29T10:58:34.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.311+0000 7fc8417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.311+0000 7fc8417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:34.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.634+0000 7fc849be3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.634+0000 7fc849be3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.634+0000 7fc849be3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:58:34.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.634+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc824005330 con 0x7fc83c0c9550 2024-03-29T10:58:34.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.635+0000 7fc8417fa640 1 -- 172.21.15.45:0/42004222 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fc830052300 con 0x7fc83c0c9550 2024-03-29T10:58:34.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.635+0000 7fc8417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.635+0000 7fc8417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:34.616 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:34.616 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 10 librados: watch_flush enter 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 10 librados: watch_flush exit 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 10 client.4468.objecter successfully canceled tick 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 >> v1:172.21.15.116:6816/3290579615 conn(0x7fc82806dab0 legacy=0x7fc82806ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 10 monclient: shutdown 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 >> v1:172.21.15.116:6789/0 conn(0x7fc83c0c9550 legacy=0x7fc83c0be190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 shutdown_connections 2024-03-29T10:58:34.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 >> 172.21.15.45:0/42004222 conn(0x7fc83c0058f0 msgr2=0x7fc83c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.638+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 shutdown_connections 2024-03-29T10:58:34.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.639+0000 7fc849be3640 1 -- 172.21.15.45:0/42004222 wait complete. 2024-03-29T10:58:34.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:34.639+0000 7fc849be3640 1 librados: shutdown 2024-03-29T10:58:34.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:41.443 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.605+0000 7f538147d640 10 monclient: get_monmap_and_config 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.605+0000 7f538147d640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.605+0000 7f538147d640 10 monclient: monmap: 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:41.606919+0000 2024-03-29T10:58:41.586 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:41.606919+0000 2024-03-29T10:58:41.587 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.587 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.587 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:41.587 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:41.587 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:41.587 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.605+0000 7f538147d640 1 Processor -- start 2024-03-29T10:58:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.606+0000 7f538147d640 1 -- start start 2024-03-29T10:58:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.606+0000 7f538147d640 10 monclient: init 2024-03-29T10:58:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.606+0000 7f538147d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.606+0000 7f538147d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 10 monclient(hunting): picked mon.noname-b con 0x7f537c1422b0 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 10 monclient(hunting): picked mon.noname-a con 0x7f537c142690 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 10 monclient(hunting): picked mon.noname-c con 0x7f537c0b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f537affd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f537c1422b0 0x7f537c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55442/0 (socket says 172.21.15.45:55442) 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f537affd640 1 -- 172.21.15.45:0/1681360198 learned_addr learned my addr 172.21.15.45:0/1681360198 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c142c80 con 0x7f537c142690 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0aae40 con 0x7f537c0b17d0 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c13fbd0 con 0x7f537c1422b0 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f538147d640 10 monclient(hunting): authenticate will time out at 1195.877392s 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.607+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519864619 0 0) 0x7f537c13fbd0 con 0x7f537c1422b0 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5364003520 con 0x7f537c1422b0 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738316899 0 0) 0x7f537c0aae40 con 0x7f537c0b17d0 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c13fbd0 con 0x7f537c0b17d0 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171178778 0 0) 0x7f537c142c80 con 0x7f537c142690 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:58:41.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c0aae40 con 0x7f537c142690 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028006102 0 0) 0x7f537c13fbd0 con 0x7f537c0b17d0 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 >> v1:172.21.15.45:6790/0 conn(0x7f537c1422b0 legacy=0x7f537c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 >> v1:172.21.15.45:6789/0 conn(0x7f537c142690 legacy=0x7f537c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 monclient: found mon.noname-c 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c151940 con 0x7f537c0b17d0 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f53740015f0 con 0x7f537c0b17d0 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f5379ffb640 10 monclient: dump: 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f538147d640 5 monclient: authenticate success, global_id 4471 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.608+0000 7f538147d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5374001750 con 0x7f537c0b17d0 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f5379ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f5379ffb640 1 -- 172.21.15.45:0/1681360198 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5374001ae0 con 0x7f537c0b17d0 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f5379ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f5379ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f5379ffb640 10 monclient: dump: 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:41.604 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f538147d640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f538147d640 4 set_mon_vals no callback set 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f538147d640 10 monclient: shutdown 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f538147d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.609+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 >> v1:172.21.15.116:6789/0 conn(0x7f537c0b17d0 legacy=0x7f537c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.610+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 shutdown_connections 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.610+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 >> 172.21.15.45:0/1681360198 conn(0x7f537c066a00 msgr2=0x7f537c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.610+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 shutdown_connections 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.610+0000 7f538147d640 1 -- 172.21.15.45:0/1681360198 wait complete. 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 10 monclient: monmap: 2024-03-29T10:58:41.605 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 1 librados: starting msgr at 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 1 librados: starting objecter 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 1 Processor -- start 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 1 -- start start 2024-03-29T10:58:41.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 1 librados: setting wanted keys 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 1 librados: calling monclient init 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.611+0000 7f538147d640 10 monclient: init 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.612+0000 7f538147d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.612+0000 7f538147d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.612+0000 7f538147d640 10 monclient(hunting): picked mon.noname-b con 0x7f537c0b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.612+0000 7f538147d640 10 monclient(hunting): picked mon.noname-a con 0x7f537c1422b0 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f537affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f537c0b17d0 0x7f537c0b0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:44830/0 (socket says 172.21.15.45:44830) 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f537a7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f537c1422b0 0x7f537c0adea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50072/0 (socket says 172.21.15.45:50072) 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f538147d640 10 monclient(hunting): picked mon.noname-c con 0x7f537c142690 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f537a7fc640 1 -- 172.21.15.45:0/1769604523 learned_addr learned my addr 172.21.15.45:0/1769604523 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f538147d640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0b14f0 con 0x7f537c1422b0 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f538147d640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0b0920 con 0x7f537c0b17d0 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f53637fe640 10 client.?.objecter ms_handle_connect 0x7f537c1422b0 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f53637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f53637fe640 10 client.?.objecter ms_handle_connect 0x7f537c0b17d0 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f53637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f538147d640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c12fb00 con 0x7f537c142690 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f538147d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f538147d640 10 monclient(hunting): authenticate will time out at 1195.883424s 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f53637fe640 10 client.?.objecter ms_handle_connect 0x7f537c142690 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.613+0000 7f53637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176757228 0 0) 0x7f537c0b14f0 con 0x7f537c1422b0 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5368004eb0 con 0x7f537c1422b0 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010038536 0 0) 0x7f537c12fb00 con 0x7f537c142690 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c0b14f0 con 0x7f537c142690 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824393703 0 0) 0x7f537c0b0920 con 0x7f537c0b17d0 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c12fb00 con 0x7f537c0b17d0 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2511486511 0 0) 0x7f537c0b14f0 con 0x7f537c142690 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f537c0b0920 con 0x7f537c142690 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f53740018c0 con 0x7f537c142690 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.614+0000 7f53637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:41.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient(hunting): dump: 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866027074 0 0) 0x7f5368004eb0 con 0x7f537c1422b0 2024-03-29T10:58:41.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f537c0b14f0 con 0x7f537c1422b0 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f53700019d0 con 0x7f537c1422b0 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient(hunting): dump: 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3794180765 0 0) 0x7f537c0b0920 con 0x7f537c142690 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 >> v1:172.21.15.116:6789/0 conn(0x7f537c0b17d0 legacy=0x7f537c0b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 >> v1:172.21.15.45:6789/0 conn(0x7f537c1422b0 legacy=0x7f537c0adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 monclient: found mon.c 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c0b0500 con 0x7f537c142690 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 10 monclient: _renew_subs 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:41.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537c0ae5b0 con 0x7f537c142690 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 10 monclient: _renew_subs 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c0af030 con 0x7f537c142690 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f538147d640 1 librados: init done 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5374001b40 con 0x7f537c142690 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.615+0000 7f53797fa640 4 set_mon_vals no callback set 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.617+0000 7f53617fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.618+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5374003a40 con 0x7f537c142690 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.618+0000 7f53637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.618+0000 7f53637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.618+0000 7f53637fe640 10 monclient: dump: 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:41.613 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.618+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f5374019a10 con 0x7f537c142690 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53617fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f5374083920 con 0x7f537c142690 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53637fe640 10 client.4478.objecter ms_dispatch 0x7f537c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:41.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53637fe640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53637fe640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53637fe640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5354005080 con 0x7f537c142690 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f537affd640 10 monclient: get_auth_request con 0x7f5368076960 auth_method 0 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.619+0000 7f53637fe640 10 client.4478.objecter ms_handle_connect 0x7f5368076960 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.627+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f537404f2c0 con 0x7f537c142690 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.627+0000 7f53637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.627+0000 7f53637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:41.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.996+0000 7f53617fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:41.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.996+0000 7f53617fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:41.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.996+0000 7f53617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:41.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.996+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5354005330 con 0x7f537c142690 2024-03-29T10:58:41.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.997+0000 7f53637fe640 1 -- 172.21.15.45:0/1769604523 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f53740511f0 con 0x7f537c142690 2024-03-29T10:58:41.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.997+0000 7f53637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:41.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:41.997+0000 7f53637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:41.978 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:41.978 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:41.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.000+0000 7f53617fa640 10 librados: watch_flush enter 2024-03-29T10:58:41.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.000+0000 7f53617fa640 10 librados: watch_flush exit 2024-03-29T10:58:41.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.000+0000 7f53617fa640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T10:58:41.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.000+0000 7f53617fa640 10 client.4478.objecter successfully canceled tick 2024-03-29T10:58:41.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 >> v1:172.21.15.116:6816/3290579615 conn(0x7f5368076960 legacy=0x7f5368078e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 10 monclient: shutdown 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 >> v1:172.21.15.45:6790/0 conn(0x7f537c142690 legacy=0x7f537c1e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 shutdown_connections 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 >> 172.21.15.45:0/1769604523 conn(0x7f537c066a00 msgr2=0x7f537c13d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 shutdown_connections 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 -- 172.21.15.45:0/1769604523 wait complete. 2024-03-29T10:58:41.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:42.001+0000 7f53617fa640 1 librados: shutdown 2024-03-29T10:58:41.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:48.804 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.972+0000 7fe6daf74640 10 monclient: get_monmap_and_config 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.972+0000 7fe6daf74640 10 monclient: build_initial_monmap 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.972+0000 7fe6daf74640 10 monclient: monmap: 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:48.973574+0000 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:48.973574+0000 2024-03-29T10:58:48.953 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:48.954 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:48.954 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:48.954 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:48.954 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:48.954 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.973+0000 7fe6daf74640 1 Processor -- start 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.973+0000 7fe6daf74640 1 -- start start 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.973+0000 7fe6daf74640 10 monclient: init 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 10 monclient(hunting): picked mon.noname-b con 0x7fe6d414a400 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:48.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 10 monclient(hunting): picked mon.noname-c con 0x7fe6d40aa500 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:48.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 10 monclient(hunting): picked mon.noname-a con 0x7fe6d4149800 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:48.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d40aca50 con 0x7fe6d4149800 2024-03-29T10:58:48.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.974+0000 7fe6daf74640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d4149cb0 con 0x7fe6d40aa500 2024-03-29T10:58:48.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.975+0000 7fe6daf74640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d40acc80 con 0x7fe6d414a400 2024-03-29T10:58:48.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.975+0000 7fe6daf74640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:48.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.975+0000 7fe6daf74640 10 monclient(hunting): authenticate will time out at 1203.244001s 2024-03-29T10:58:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d9f72640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6d414a400 0x7fe6d40aac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40599/0 (socket says 172.21.15.45:40599) 2024-03-29T10:58:48.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d9f72640 1 -- 172.21.15.45:0/845159078 learned_addr learned my addr 172.21.15.45:0/845159078 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169263532 0 0) 0x7fe6d40acc80 con 0x7fe6d414a400 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d8f70640 10 monclient(hunting): _negotiate 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d8f70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d8f70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d8f70640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.978+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c0003520 con 0x7fe6d414a400 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.979+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054317004 0 0) 0x7fe6c0003520 con 0x7fe6d414a400 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.979+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 >> v1:172.21.15.116:6789/0 conn(0x7fe6d40aa500 legacy=0x7fe6d40ad160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 >> v1:172.21.15.45:6789/0 conn(0x7fe6d4149800 legacy=0x7fe6d4151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:48.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: _finish_hunting 0 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 1 monclient: found mon.noname-b 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40ac350 con 0x7fe6d414a400 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: _finish_auth 0 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: _check_auth_tickets 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6daf74640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6daf74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6d0001690 con 0x7fe6d414a400 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.980+0000 7fe6d8f70640 10 monclient: dump: 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:48.963 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.981+0000 7fe6daf74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.981+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6d0001860 con 0x7fe6d414a400 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.981+0000 7fe6d8f70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.981+0000 7fe6daf74640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.981+0000 7fe6daf74640 4 set_mon_vals no callback set 2024-03-29T10:58:48.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6d8f70640 1 -- 172.21.15.45:0/845159078 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6d0001cb0 con 0x7fe6d414a400 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6d8f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6d8f70640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6d8f70640 10 monclient: dump: 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:48.965 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:48.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6daf74640 10 monclient: shutdown 2024-03-29T10:58:48.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6daf74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:48.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.982+0000 7fe6daf74640 1 -- 172.21.15.45:0/845159078 >> v1:172.21.15.45:6790/0 conn(0x7fe6d414a400 legacy=0x7fe6d40aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.983+0000 7fe6daf74640 1 -- 172.21.15.45:0/845159078 shutdown_connections 2024-03-29T10:58:48.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.983+0000 7fe6daf74640 1 -- 172.21.15.45:0/845159078 >> 172.21.15.45:0/845159078 conn(0x7fe6d40b3a30 msgr2=0x7fe6d40b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.983+0000 7fe6daf74640 1 -- 172.21.15.45:0/845159078 shutdown_connections 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.985+0000 7fe6daf74640 1 -- 172.21.15.45:0/845159078 wait complete. 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.986+0000 7fe6daf74640 10 monclient: build_initial_monmap 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.986+0000 7fe6daf74640 10 monclient: monmap: 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:48.967 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 1 librados: starting msgr at 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 1 librados: starting objecter 2024-03-29T10:58:48.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 1 Processor -- start 2024-03-29T10:58:48.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 1 -- start start 2024-03-29T10:58:48.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 1 librados: setting wanted keys 2024-03-29T10:58:48.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 1 librados: calling monclient init 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.987+0000 7fe6daf74640 10 monclient: init 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient(hunting): picked mon.noname-a con 0x7fe6d40aa500 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient(hunting): picked mon.noname-c con 0x7fe6d4149800 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient(hunting): picked mon.noname-b con 0x7fe6d4145080 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d4147240 con 0x7fe6d40aa500 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d41388f0 con 0x7fe6d4145080 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d4146540 con 0x7fe6d4149800 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6daf74640 10 monclient(hunting): authenticate will time out at 1203.257394s 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6d9f72640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe6d40aa500 0x7fe6d4145e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38004/0 (socket says 172.21.15.45:38004) 2024-03-29T10:58:48.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.988+0000 7fe6d9f72640 1 -- 172.21.15.45:0/271847122 learned_addr learned my addr 172.21.15.45:0/271847122 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 10 client.?.objecter ms_handle_connect 0x7fe6d40aa500 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249773540 0 0) 0x7fe6d4147240 con 0x7fe6d40aa500 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.989+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b8004eb0 con 0x7fe6d40aa500 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181274515 0 0) 0x7fe6b8004eb0 con 0x7fe6d40aa500 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6d4147240 con 0x7fe6d40aa500 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6d0001b70 con 0x7fe6d40aa500 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:48.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 10 monclient(hunting): dump: 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:48.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3133221300 0 0) 0x7fe6d4147240 con 0x7fe6d40aa500 2024-03-29T10:58:48.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.991+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 >> v1:172.21.15.45:6790/0 conn(0x7fe6d4149800 legacy=0x7fe6d4138d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 >> v1:172.21.15.116:6789/0 conn(0x7fe6d4145080 legacy=0x7fe6d4139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 1 monclient: found mon.a 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d413eeb0 con 0x7fe6d40aa500 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 5 monclient: authenticate success, global_id 4455 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 10 monclient: _renew_subs 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d41e6db0 con 0x7fe6d40aa500 2024-03-29T10:58:48.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 10 monclient: _renew_subs 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d41e7180 con 0x7fe6d40aa500 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6daf74640 1 librados: init done 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6d0004bf0 con 0x7fe6d40aa500 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6d0004f00 con 0x7fe6d40aa500 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.992+0000 7fe6ca7fc640 10 monclient: dump: 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.979 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.993+0000 7fe6cbfff640 4 set_mon_vals no callback set 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.994+0000 7fe6daf74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.995+0000 7fe6daf74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.995+0000 7fe6daf74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:48.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.995+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6bc005080 con 0x7fe6d40aa500 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.997+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe6d001aed0 con 0x7fe6d40aa500 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.998+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fe6d0084ba0 con 0x7fe6d40aa500 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.998+0000 7fe6ca7fc640 10 client.4455.objecter ms_dispatch 0x7fe6d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.998+0000 7fe6ca7fc640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.998+0000 7fe6ca7fc640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.998+0000 7fe6d9771640 10 monclient: get_auth_request con 0x7fe6b806dc30 auth_method 0 2024-03-29T10:58:48.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:48.998+0000 7fe6ca7fc640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T10:58:48.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.001+0000 7fe6ca7fc640 10 client.4455.objecter ms_handle_connect 0x7fe6b806dc30 2024-03-29T10:58:48.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.009+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6d0050480 con 0x7fe6d40aa500 2024-03-29T10:58:48.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.009+0000 7fe6ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:48.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.009+0000 7fe6ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:49.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.338+0000 7fe6daf74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.338+0000 7fe6daf74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.338+0000 7fe6daf74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:58:49.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.338+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6bc005330 con 0x7fe6d40aa500 2024-03-29T10:58:49.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.339+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/271847122 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fe6d00523b0 con 0x7fe6d40aa500 2024-03-29T10:58:49.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.339+0000 7fe6ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.339+0000 7fe6ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:49.322 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:49.323 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:49.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 10 librados: watch_flush enter 2024-03-29T10:58:49.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 10 librados: watch_flush exit 2024-03-29T10:58:49.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T10:58:49.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 10 client.4455.objecter successfully canceled tick 2024-03-29T10:58:49.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe6b806dc30 legacy=0x7fe6b80700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 10 monclient: shutdown 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 >> v1:172.21.15.45:6789/0 conn(0x7fe6d40aa500 legacy=0x7fe6d4145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 shutdown_connections 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.346+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 >> 172.21.15.45:0/271847122 conn(0x7fe6d40b3a30 msgr2=0x7fe6d40ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.347+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 shutdown_connections 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.347+0000 7fe6daf74640 1 -- 172.21.15.45:0/271847122 wait complete. 2024-03-29T10:58:49.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:49.347+0000 7fe6daf74640 1 librados: shutdown 2024-03-29T10:58:49.344 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:56.151 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:56.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.306+0000 7fd7022ba640 10 monclient: get_monmap_and_config 2024-03-29T10:58:56.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.306+0000 7fd7022ba640 10 monclient: build_initial_monmap 2024-03-29T10:58:56.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.306+0000 7fd7022ba640 10 monclient: monmap: 2024-03-29T10:58:56.287 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:56.288 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:56.288 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:58:56.307524+0000 2024-03-29T10:58:56.288 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:58:56.307524+0000 2024-03-29T10:58:56.288 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:56.288 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:56.288 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:56.289 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:58:56.289 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:58:56.289 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.307+0000 7fd7022ba640 1 Processor -- start 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.307+0000 7fd7022ba640 1 -- start start 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.307+0000 7fd7022ba640 10 monclient: init 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient(hunting): picked mon.noname-b con 0x7fd6fc0b4790 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient(hunting): picked mon.noname-a con 0x7fd6fc0aea90 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient(hunting): picked mon.noname-c con 0x7fd6fc0b39f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0b0c60 con 0x7fd6fc0aea90 2024-03-29T10:58:56.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0af020 con 0x7fd6fc0b39f0 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc13fbd0 con 0x7fd6fc0b4790 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd7022ba640 10 monclient(hunting): authenticate will time out at 1210.577591s 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd6fbfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd6fc0b4790 0x7fd6fc0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57230/0 (socket says 172.21.15.45:57230) 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.308+0000 7fd6fbfff640 1 -- 172.21.15.45:0/1053794021 learned_addr learned my addr 172.21.15.45:0/1053794021 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403827812 0 0) 0x7fd6fc13fbd0 con 0x7fd6fc0b4790 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e4003520 con 0x7fd6fc0b4790 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260277472 0 0) 0x7fd6e4003520 con 0x7fd6fc0b4790 2024-03-29T10:58:56.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 >> v1:172.21.15.116:6789/0 conn(0x7fd6fc0b39f0 legacy=0x7fd6fc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:56.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.309+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 >> v1:172.21.15.45:6789/0 conn(0x7fd6fc0aea90 legacy=0x7fd6fc0af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:56.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:56.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 1 monclient: found mon.noname-b 2024-03-29T10:58:56.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:56.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:58:56.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc13fd60 con 0x7fd6fc0b4790 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd6ec001690 con 0x7fd6fc0b4790 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: dump: 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:56.296 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6ec0018f0 con 0x7fd6fc0b4790 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 1 -- 172.21.15.45:0/1053794021 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd6ec001d40 con 0x7fd6fc0b4790 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd6faffd640 10 monclient: dump: 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:56.297 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.312+0000 7fd7022ba640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.313+0000 7fd7022ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.313+0000 7fd7022ba640 4 set_mon_vals no callback set 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.314+0000 7fd7022ba640 10 monclient: shutdown 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.314+0000 7fd7022ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.314+0000 7fd7022ba640 1 -- 172.21.15.45:0/1053794021 >> v1:172.21.15.45:6790/0 conn(0x7fd6fc0b4790 legacy=0x7fd6fc0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.314+0000 7fd7022ba640 1 -- 172.21.15.45:0/1053794021 shutdown_connections 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.315+0000 7fd7022ba640 1 -- 172.21.15.45:0/1053794021 >> 172.21.15.45:0/1053794021 conn(0x7fd6fc066a00 msgr2=0x7fd6fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.315+0000 7fd7022ba640 1 -- 172.21.15.45:0/1053794021 shutdown_connections 2024-03-29T10:58:56.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.315+0000 7fd7022ba640 1 -- 172.21.15.45:0/1053794021 wait complete. 2024-03-29T10:58:56.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.316+0000 7fd7022ba640 10 monclient: build_initial_monmap 2024-03-29T10:58:56.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.316+0000 7fd7022ba640 10 monclient: monmap: 2024-03-29T10:58:56.313 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:58:56.313 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:56.313 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:58:56.313 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.316+0000 7fd7022ba640 1 librados: starting msgr at 2024-03-29T10:58:56.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.316+0000 7fd7022ba640 1 librados: starting objecter 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.316+0000 7fd7022ba640 1 Processor -- start 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.317+0000 7fd7022ba640 1 -- start start 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.317+0000 7fd7022ba640 1 librados: setting wanted keys 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.317+0000 7fd7022ba640 1 librados: calling monclient init 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.317+0000 7fd7022ba640 10 monclient: init 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.318+0000 7fd7022ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.318+0000 7fd7022ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.318+0000 7fd7022ba640 10 monclient(hunting): picked mon.noname-b con 0x7fd6fc0b39f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.318+0000 7fd7022ba640 10 monclient(hunting): picked mon.noname-a con 0x7fd6fc0b4790 addr v1:172.21.15.45:6789/0 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd7022ba640 10 monclient(hunting): picked mon.noname-c con 0x7fd6fc0718c0 addr v1:172.21.15.45:6790/0 2024-03-29T10:58:56.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd7022ba640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0a9870 con 0x7fd6fc0b4790 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd6fbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd6fc0b39f0 0x7fd6fc070990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:39086/0 (socket says 172.21.15.45:39086) 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd7022ba640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc071150 con 0x7fd6fc0b39f0 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd6fbfff640 1 -- 172.21.15.45:0/2644137954 learned_addr learned my addr 172.21.15.45:0/2644137954 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0a9aa0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd7022ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd7022ba640 10 monclient(hunting): authenticate will time out at 1210.588582s 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd6f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd6fc0718c0 2024-03-29T10:58:56.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd6f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd6f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd6fc0b39f0 2024-03-29T10:58:56.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.319+0000 7fd6f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787348232 0 0) 0x7fd6fc0a9aa0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e8004eb0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242737318 0 0) 0x7fd6e8004eb0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6fc0a9aa0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd6f4001890 con 0x7fd6fc0718c0 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:56.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): dump: 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:56.319 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.320+0000 7fd6f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.321+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4101896452 0 0) 0x7fd6fc0a9aa0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.321+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 >> v1:172.21.15.116:6789/0 conn(0x7fd6fc0b39f0 legacy=0x7fd6fc070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.321+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 >> v1:172.21.15.45:6789/0 conn(0x7fd6fc0b4790 legacy=0x7fd6fc06de90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.322+0000 7fd6f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 1 monclient: found mon.c 2024-03-29T10:58:56.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0a9d80 con 0x7fd6fc0718c0 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd6f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 10 monclient: _renew_subs 2024-03-29T10:58:56.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc0aa060 con 0x7fd6fc0718c0 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 10 monclient: _renew_subs 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc06e5a0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.323+0000 7fd7022ba640 1 librados: init done 2024-03-29T10:58:56.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.324+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6f4001e20 con 0x7fd6fc0718c0 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.324+0000 7fd6f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.324+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fd6f4003c10 con 0x7fd6fc0718c0 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.325+0000 7fd6fa7fc640 4 set_mon_vals no callback set 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.325+0000 7fd6f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.325+0000 7fd6f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.325+0000 7fd6f8ff9640 10 monclient: dump: 2024-03-29T10:58:56.323 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:58:56.324 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:58:56.324 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.324 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:58:56.324 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.325+0000 7fd7022ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.326+0000 7fd7022ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.326+0000 7fd7022ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:56.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.326+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6dc005080 con 0x7fd6fc0718c0 2024-03-29T10:58:56.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fd6f4019be0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6fbfff640 10 monclient: get_auth_request con 0x7fd6e806dca0 auth_method 0 2024-03-29T10:58:56.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fd6f4053950 con 0x7fd6fc0718c0 2024-03-29T10:58:56.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6f8ff9640 10 client.4487.objecter ms_dispatch 0x7fd6fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:56.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6f8ff9640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:56.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6f8ff9640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:56.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.327+0000 7fd6f8ff9640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T10:58:56.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.328+0000 7fd6f8ff9640 10 client.4487.objecter ms_handle_connect 0x7fd6e806dca0 2024-03-29T10:58:56.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.335+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6f404fbf0 con 0x7fd6fc0718c0 2024-03-29T10:58:56.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.335+0000 7fd6f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.335+0000 7fd6f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:56.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.645+0000 7fd7022ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.645+0000 7fd7022ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.645+0000 7fd7022ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:58:56.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.645+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6dc005330 con 0x7fd6fc0718c0 2024-03-29T10:58:56.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.646+0000 7fd6f8ff9640 1 -- 172.21.15.45:0/2644137954 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fd6f404fd90 con 0x7fd6fc0718c0 2024-03-29T10:58:56.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.646+0000 7fd6f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.646+0000 7fd6f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:56.626 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:58:56.627 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:56.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 10 librados: watch_flush enter 2024-03-29T10:58:56.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 10 librados: watch_flush exit 2024-03-29T10:58:56.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 10 client.4487.objecter successfully canceled tick 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 >> v1:172.21.15.116:6816/3290579615 conn(0x7fd6e806dca0 legacy=0x7fd6e8070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 10 monclient: shutdown 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 >> v1:172.21.15.45:6790/0 conn(0x7fd6fc0718c0 legacy=0x7fd6fc1e6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 shutdown_connections 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 >> 172.21.15.45:0/2644137954 conn(0x7fd6fc066a00 msgr2=0x7fd6fc0b6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 shutdown_connections 2024-03-29T10:58:56.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.649+0000 7fd7022ba640 1 -- 172.21.15.45:0/2644137954 wait complete. 2024-03-29T10:58:56.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:58:56.651+0000 7fd7022ba640 1 librados: shutdown 2024-03-29T10:58:56.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:03.452 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:03.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.620+0000 7f72eab6c640 10 monclient: get_monmap_and_config 2024-03-29T10:59:03.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.620+0000 7f72eab6c640 10 monclient: build_initial_monmap 2024-03-29T10:59:03.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.620+0000 7f72eab6c640 10 monclient: monmap: 2024-03-29T10:59:03.602 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:03.602 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:03.602 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:03.621279+0000 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:03.621279+0000 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.621+0000 7f72eab6c640 1 Processor -- start 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.621+0000 7f72eab6c640 1 -- start start 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.621+0000 7f72eab6c640 10 monclient: init 2024-03-29T10:59:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient(hunting): picked mon.noname-b con 0x7f72e40a9930 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient(hunting): picked mon.noname-c con 0x7f72e4149dc0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient(hunting): picked mon.noname-a con 0x7f72e414ff40 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40aae60 con 0x7f72e414ff40 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40b4310 con 0x7f72e4149dc0 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40afdd0 con 0x7f72e40a9930 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.622+0000 7f72eab6c640 10 monclient(hunting): authenticate will time out at 1217.891917s 2024-03-29T10:59:03.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.623+0000 7f72e8908640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f72e40a9930 0x7f72e4143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:41640/0 (socket says 172.21.15.45:41640) 2024-03-29T10:59:03.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.623+0000 7f72e8908640 1 -- 172.21.15.45:0/3088814514 learned_addr learned my addr 172.21.15.45:0/3088814514 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.626+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561743900 0 0) 0x7f72e40aae60 con 0x7f72e414ff40 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72cc003520 con 0x7f72e414ff40 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056241338 0 0) 0x7f72e40afdd0 con 0x7f72e40a9930 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e40aae60 con 0x7f72e40a9930 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1949472931 0 0) 0x7f72cc003520 con 0x7f72e414ff40 2024-03-29T10:59:03.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 >> v1:172.21.15.45:6790/0 conn(0x7f72e40a9930 legacy=0x7f72e4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 >> v1:172.21.15.116:6789/0 conn(0x7f72e4149dc0 legacy=0x7f72e414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 monclient: found mon.noname-a 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e414a950 con 0x7f72e414ff40 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f72dc001a10 con 0x7f72e414ff40 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: dump: 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:03.609 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72dc001c30 con 0x7f72e414ff40 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 1 -- 172.21.15.45:0/3088814514 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f72dc001fc0 con 0x7f72e414ff40 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72e37fe640 10 monclient: dump: 2024-03-29T10:59:03.610 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:03.611 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72eab6c640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72eab6c640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.627+0000 7f72eab6c640 4 set_mon_vals no callback set 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 10 monclient: shutdown 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 1 -- 172.21.15.45:0/3088814514 >> v1:172.21.15.45:6789/0 conn(0x7f72e414ff40 legacy=0x7f72e4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 1 -- 172.21.15.45:0/3088814514 shutdown_connections 2024-03-29T10:59:03.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 1 -- 172.21.15.45:0/3088814514 >> 172.21.15.45:0/3088814514 conn(0x7f72e40b35e0 msgr2=0x7f72e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:03.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 1 -- 172.21.15.45:0/3088814514 shutdown_connections 2024-03-29T10:59:03.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.630+0000 7f72eab6c640 1 -- 172.21.15.45:0/3088814514 wait complete. 2024-03-29T10:59:03.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.631+0000 7f72eab6c640 10 monclient: build_initial_monmap 2024-03-29T10:59:03.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.631+0000 7f72eab6c640 10 monclient: monmap: 2024-03-29T10:59:03.614 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:03.614 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.631+0000 7f72eab6c640 1 librados: starting msgr at 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.631+0000 7f72eab6c640 1 librados: starting objecter 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.632+0000 7f72eab6c640 1 Processor -- start 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.632+0000 7f72eab6c640 1 -- start start 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.632+0000 7f72eab6c640 1 librados: setting wanted keys 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.632+0000 7f72eab6c640 1 librados: calling monclient init 2024-03-29T10:59:03.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.632+0000 7f72eab6c640 10 monclient: init 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient(hunting): picked mon.noname-c con 0x7f72e40a9930 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient(hunting): picked mon.noname-b con 0x7f72e4149dc0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient(hunting): picked mon.noname-a con 0x7f72e414ff40 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e413b8c0 con 0x7f72e414ff40 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e413bba0 con 0x7f72e4149dc0 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e413be80 con 0x7f72e40a9930 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72eab6c640 10 monclient(hunting): authenticate will time out at 1217.903393s 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72e8908640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f72e40a9930 0x7f72e4148380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:52842/0 (socket says 172.21.15.45:52842) 2024-03-29T10:59:03.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72e8908640 1 -- 172.21.15.45:0/3308100479 learned_addr learned my addr 172.21.15.45:0/3308100479 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:03.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72e17fa640 10 client.?.objecter ms_handle_connect 0x7f72e40a9930 2024-03-29T10:59:03.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.634+0000 7f72e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.637+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796235394 0 0) 0x7f72e413be80 con 0x7f72e40a9930 2024-03-29T10:59:03.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.637+0000 7f72e17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.637+0000 7f72e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.637+0000 7f72e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.637+0000 7f72e17fa640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:59:03.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.637+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d0004eb0 con 0x7f72e40a9930 2024-03-29T10:59:03.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343729018 0 0) 0x7f72d0004eb0 con 0x7f72e40a9930 2024-03-29T10:59:03.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72e413be80 con 0x7f72e40a9930 2024-03-29T10:59:03.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f72d4001840 con 0x7f72e40a9930 2024-03-29T10:59:03.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:03.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 10 monclient(hunting): dump: 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760057270 0 0) 0x7f72e413be80 con 0x7f72e40a9930 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.638+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 >> v1:172.21.15.116:6789/0 conn(0x7f72e4149dc0 legacy=0x7f72e4148e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 >> v1:172.21.15.45:6789/0 conn(0x7f72e414ff40 legacy=0x7f72e42000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 1 monclient: found mon.c 2024-03-29T10:59:03.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:03.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e4138b20 con 0x7f72e40a9930 2024-03-29T10:59:03.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 10 client.?.objecter ms_handle_connect 0x7f72e414ff40 2024-03-29T10:59:03.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:59:03.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 10 monclient: _renew_subs 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e40ab5c0 con 0x7f72e40a9930 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 10 monclient: _renew_subs 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e4137d50 con 0x7f72e40a9930 2024-03-29T10:59:03.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.639+0000 7f72eab6c640 1 librados: init done 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72eab6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72eab6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72eab6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e4075f10 con 0x7f72e40a9930 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72d40019a0 con 0x7f72e40a9930 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f72d4004a00 con 0x7f72e40a9930 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e17fa640 10 monclient: dump: 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:03.626 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.641+0000 7f72e27fc640 4 set_mon_vals no callback set 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.642+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f72d4004c20 con 0x7f72e40a9930 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.642+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f72d4084b10 con 0x7f72e40a9930 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.642+0000 7f72e17fa640 10 client.4493.objecter ms_dispatch 0x7f72e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.642+0000 7f72e17fa640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.642+0000 7f72e17fa640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.643+0000 7f72e17fa640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T10:59:03.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.645+0000 7f72e3fff640 10 monclient: get_auth_request con 0x7f72d006dc50 auth_method 0 2024-03-29T10:59:03.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.646+0000 7f72e17fa640 10 client.4493.objecter ms_handle_connect 0x7f72d006dc50 2024-03-29T10:59:03.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.649+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72d40503f0 con 0x7f72e40a9930 2024-03-29T10:59:03.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.649+0000 7f72e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.649+0000 7f72e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.978+0000 7f72eab6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.978+0000 7f72eab6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.978+0000 7f72eab6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.978+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72e406ab70 con 0x7f72e40a9930 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.979+0000 7f72e17fa640 1 -- 172.21.15.45:0/3308100479 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f72d4052320 con 0x7f72e40a9930 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.979+0000 7f72e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.979+0000 7f72e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.983+0000 7f72eab6c640 10 librados: watch_flush enter 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.983+0000 7f72eab6c640 10 librados: watch_flush exit 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.983+0000 7f72eab6c640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.983+0000 7f72eab6c640 10 client.4493.objecter successfully canceled tick 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.984+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 >> v1:172.21.15.116:6816/3290579615 conn(0x7f72d006dc50 legacy=0x7f72d0070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.984+0000 7f72eab6c640 10 monclient: shutdown 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.984+0000 7f72eab6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.984+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 >> v1:172.21.15.45:6790/0 conn(0x7f72e40a9930 legacy=0x7f72e4148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.985+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 shutdown_connections 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.985+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 >> 172.21.15.45:0/3308100479 conn(0x7f72e40b35e0 msgr2=0x7f72e40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.985+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 shutdown_connections 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.985+0000 7f72eab6c640 1 -- 172.21.15.45:0/3308100479 wait complete. 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:03.985+0000 7f72eab6c640 1 librados: shutdown 2024-03-29T10:59:03.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:10.790 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:10.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.959+0000 7f80e95ec640 10 monclient: get_monmap_and_config 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.959+0000 7f80e95ec640 10 monclient: build_initial_monmap 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.959+0000 7f80e95ec640 10 monclient: monmap: 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:10.960333+0000 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:10.960333+0000 2024-03-29T10:59:10.940 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:10.941 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:10.941 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:10.941 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:10.941 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:10.941 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:10.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.961+0000 7f80e95ec640 1 Processor -- start 2024-03-29T10:59:10.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.962+0000 7f80e95ec640 1 -- start start 2024-03-29T10:59:10.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.962+0000 7f80e95ec640 10 monclient: init 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.962+0000 7f80e95ec640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.962+0000 7f80e95ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 10 monclient(hunting): picked mon.noname-c con 0x7f80dc006e30 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 10 monclient(hunting): picked mon.noname-a con 0x7f80dc035e00 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 10 monclient(hunting): picked mon.noname-b con 0x7f80dc0c9670 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc035970 con 0x7f80dc035e00 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc036280 con 0x7f80dc006e30 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc035640 con 0x7f80dc0c9670 2024-03-29T10:59:10.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:10.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e95ec640 10 monclient(hunting): authenticate will time out at 1225.232135s 2024-03-29T10:59:10.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e8deb640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f80dc0c9670 0x7f80dc0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35572/0 (socket says 172.21.15.45:35572) 2024-03-29T10:59:10.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e8deb640 1 -- 172.21.15.45:0/2532090064 learned_addr learned my addr 172.21.15.45:0/2532090064 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:10.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824002508 0 0) 0x7f80dc035640 con 0x7f80dc0c9670 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e2ffd640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.963+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4003520 con 0x7f80dc0c9670 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.964+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754276751 0 0) 0x7f80d4003520 con 0x7f80dc0c9670 2024-03-29T10:59:10.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.964+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 >> v1:172.21.15.116:6789/0 conn(0x7f80dc006e30 legacy=0x7f80dc0cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:10.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 >> v1:172.21.15.45:6789/0 conn(0x7f80dc035e00 legacy=0x7f80dc0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc034e20 con 0x7f80dc0c9670 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80d8001690 con 0x7f80dc0c9670 2024-03-29T10:59:10.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: dump: 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:10.952 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e95ec640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e95ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d80019a0 con 0x7f80dc0c9670 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 1 -- 172.21.15.45:0/2532090064 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80d8001d30 con 0x7f80dc0c9670 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e2ffd640 10 monclient: dump: 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:10.953 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e95ec640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:10.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.967+0000 7f80e95ec640 4 set_mon_vals no callback set 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 10 monclient: shutdown 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 1 -- 172.21.15.45:0/2532090064 >> v1:172.21.15.45:6790/0 conn(0x7f80dc0c9670 legacy=0x7f80dc0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 1 -- 172.21.15.45:0/2532090064 shutdown_connections 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 1 -- 172.21.15.45:0/2532090064 >> 172.21.15.45:0/2532090064 conn(0x7f80dc005c00 msgr2=0x7f80dc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 1 -- 172.21.15.45:0/2532090064 shutdown_connections 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.968+0000 7f80e95ec640 1 -- 172.21.15.45:0/2532090064 wait complete. 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.969+0000 7f80e95ec640 10 monclient: build_initial_monmap 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.969+0000 7f80e95ec640 10 monclient: monmap: 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:10.955 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.970+0000 7f80e95ec640 1 librados: starting msgr at 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.970+0000 7f80e95ec640 1 librados: starting objecter 2024-03-29T10:59:10.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.970+0000 7f80e95ec640 1 Processor -- start 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.971+0000 7f80e95ec640 1 -- start start 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.971+0000 7f80e95ec640 1 librados: setting wanted keys 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.971+0000 7f80e95ec640 1 librados: calling monclient init 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.971+0000 7f80e95ec640 10 monclient: init 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.972+0000 7f80e95ec640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.972+0000 7f80e95ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.972+0000 7f80e95ec640 10 monclient(hunting): picked mon.noname-c con 0x7f80dc006e30 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.972+0000 7f80e3fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f80dc006e30 0x7f80dc0336a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35768/0 (socket says 172.21.15.45:35768) 2024-03-29T10:59:10.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e3fff640 1 -- 172.21.15.45:0/218893816 learned_addr learned my addr 172.21.15.45:0/218893816 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 10 monclient(hunting): picked mon.noname-a con 0x7f80dc035e00 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 10 monclient(hunting): picked mon.noname-b con 0x7f80dc0c9670 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc0331f0 con 0x7f80dc035e00 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc0c13f0 con 0x7f80dc0c9670 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc02dc40 con 0x7f80dc006e30 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e95ec640 10 monclient(hunting): authenticate will time out at 1225.242557s 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 10 client.?.objecter ms_handle_connect 0x7f80dc006e30 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879588468 0 0) 0x7f80dc02dc40 con 0x7f80dc006e30 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:10.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c8004eb0 con 0x7f80dc006e30 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.973+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674927717 0 0) 0x7f80c8004eb0 con 0x7f80dc006e30 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80dc02dc40 con 0x7f80dc006e30 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80e404fdd0 con 0x7f80dc006e30 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:10.965 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 206527235 0 0) 0x7f80dc02dc40 con 0x7f80dc006e30 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 >> v1:172.21.15.116:6789/0 conn(0x7f80dc0c9670 legacy=0x7f80dc0c0ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.974+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 >> v1:172.21.15.45:6789/0 conn(0x7f80dc035e00 legacy=0x7f80dc0c05d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:10.966 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 1 monclient: found mon.c 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc02df20 con 0x7f80dc006e30 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 10 monclient: _renew_subs 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80dc033db0 con 0x7f80dc006e30 2024-03-29T10:59:10.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 10 monclient: _renew_subs 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80dc0311d0 con 0x7f80dc006e30 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e95ec640 1 librados: init done 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80e404f400 con 0x7f80dc006e30 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f80e4075710 con 0x7f80dc006e30 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.976+0000 7f80e0ff9640 10 monclient: dump: 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:10.968 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.977+0000 7f80e1ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.977+0000 7f80e95ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.978+0000 7f80e95ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.978+0000 7f80e95ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:10.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.978+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c4005080 con 0x7f80dc006e30 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.978+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f80e4067bf0 con 0x7f80dc006e30 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e37fe640 10 monclient: get_auth_request con 0x7f80c806db70 auth_method 0 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f80e40ff750 con 0x7f80dc006e30 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e0ff9640 10 client.4499.objecter ms_dispatch 0x7f80e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e0ff9640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e0ff9640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e0ff9640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:59:10.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.979+0000 7f80e0ff9640 10 client.4499.objecter ms_handle_connect 0x7f80c806db70 2024-03-29T10:59:10.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.985+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80e40cb150 con 0x7f80dc006e30 2024-03-29T10:59:10.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.986+0000 7f80e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:10.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:10.986+0000 7f80e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:11.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.315+0000 7f80e95ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.315+0000 7f80e95ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.315+0000 7f80e95ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:11.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.315+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80c4005330 con 0x7f80dc006e30 2024-03-29T10:59:11.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.316+0000 7f80e0ff9640 1 -- 172.21.15.45:0/218893816 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f80e40cd080 con 0x7f80dc006e30 2024-03-29T10:59:11.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.316+0000 7f80e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.316+0000 7f80e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:11.297 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:11.297 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:11.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.319+0000 7f80e95ec640 10 librados: watch_flush enter 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.319+0000 7f80e95ec640 10 librados: watch_flush exit 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.319+0000 7f80e95ec640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.319+0000 7f80e95ec640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 >> v1:172.21.15.116:6816/3290579615 conn(0x7f80c806db70 legacy=0x7f80c8070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 10 monclient: shutdown 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:11.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 >> v1:172.21.15.45:6790/0 conn(0x7f80dc006e30 legacy=0x7f80dc0336a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 shutdown_connections 2024-03-29T10:59:11.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 >> 172.21.15.45:0/218893816 conn(0x7f80dc005c00 msgr2=0x7f80dc0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:11.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 shutdown_connections 2024-03-29T10:59:11.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.320+0000 7f80e95ec640 1 -- 172.21.15.45:0/218893816 wait complete. 2024-03-29T10:59:11.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:11.321+0000 7f80e95ec640 1 librados: shutdown 2024-03-29T10:59:11.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:18.127 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.300+0000 7f26f124a640 10 monclient: get_monmap_and_config 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.300+0000 7f26f124a640 10 monclient: build_initial_monmap 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.300+0000 7f26f124a640 10 monclient: monmap: 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:18.301754+0000 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:18.301754+0000 2024-03-29T10:59:18.281 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:18.282 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:18.282 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:18.282 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:18.282 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:18.282 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:18.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.303+0000 7f26f124a640 1 Processor -- start 2024-03-29T10:59:18.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.303+0000 7f26f124a640 1 -- start start 2024-03-29T10:59:18.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.303+0000 7f26f124a640 10 monclient: init 2024-03-29T10:59:18.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.304+0000 7f26f124a640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:18.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.304+0000 7f26f124a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:18.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.304+0000 7f26f124a640 10 monclient(hunting): picked mon.noname-c con 0x7f26ec1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:18.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.304+0000 7f26f124a640 10 monclient(hunting): picked mon.noname-b con 0x7f26ec13f650 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:18.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26ead9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f26ec1457f0 0x7f26ec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:37510/0 (socket says 172.21.15.45:37510) 2024-03-29T10:59:18.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26ead9d640 1 -- 172.21.15.45:0/3444295309 learned_addr learned my addr 172.21.15.45:0/3444295309 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:18.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26f124a640 10 monclient(hunting): picked mon.noname-a con 0x7f26ec0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:18.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec0aaef0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec145bd0 con 0x7f26ec1457f0 2024-03-29T10:59:18.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec145eb0 con 0x7f26ec13f650 2024-03-29T10:59:18.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26f124a640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.305+0000 7f26f124a640 10 monclient(hunting): authenticate will time out at 1232.574640s 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799234813 0 0) 0x7f26ec145eb0 con 0x7f26ec13f650 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d4003520 con 0x7f26ec13f650 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433311470 0 0) 0x7f26ec145bd0 con 0x7f26ec1457f0 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.309+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ec145eb0 con 0x7f26ec1457f0 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785483430 0 0) 0x7f26d4003520 con 0x7f26ec13f650 2024-03-29T10:59:18.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 >> v1:172.21.15.116:6789/0 conn(0x7f26ec1457f0 legacy=0x7f26ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 >> v1:172.21.15.45:6789/0 conn(0x7f26ec0a9d70 legacy=0x7f26ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 monclient: found mon.noname-b 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec146340 con 0x7f26ec13f650 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f26e0001690 con 0x7f26ec13f650 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: dump: 2024-03-29T10:59:18.294 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26e00017f0 con 0x7f26ec13f650 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 1 -- 172.21.15.45:0/3444295309 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f26e0001c00 con 0x7f26ec13f650 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:18.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.310+0000 7f26e9d9b640 10 monclient: dump: 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.311+0000 7f26f124a640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.311+0000 7f26f124a640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.311+0000 7f26f124a640 4 set_mon_vals no callback set 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.311+0000 7f26f124a640 10 monclient: shutdown 2024-03-29T10:59:18.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.311+0000 7f26f124a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:18.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.311+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 >> v1:172.21.15.45:6790/0 conn(0x7f26ec13f650 legacy=0x7f26ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.313+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 shutdown_connections 2024-03-29T10:59:18.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.313+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 >> 172.21.15.45:0/3444295309 conn(0x7f26ec066a00 msgr2=0x7f26ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.313+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 shutdown_connections 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.313+0000 7f26f124a640 1 -- 172.21.15.45:0/3444295309 wait complete. 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.314+0000 7f26f124a640 10 monclient: build_initial_monmap 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 10 monclient: monmap: 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:18.298 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 1 librados: starting msgr at 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 1 librados: starting objecter 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 1 Processor -- start 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 1 -- start start 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 1 librados: setting wanted keys 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 1 librados: calling monclient init 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.315+0000 7f26f124a640 10 monclient: init 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.316+0000 7f26f124a640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.316+0000 7f26f124a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:18.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.316+0000 7f26f124a640 10 monclient(hunting): picked mon.noname-c con 0x7f26ec0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 10 monclient(hunting): picked mon.noname-a con 0x7f26ec13f650 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 10 monclient(hunting): picked mon.noname-b con 0x7f26ec1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec144f80 con 0x7f26ec13f650 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec13a100 con 0x7f26ec1457f0 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec1e8470 con 0x7f26ec0a9d70 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26f124a640 10 monclient(hunting): authenticate will time out at 1232.588335s 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26ead9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f26ec0a9d70 0x7f26ec1e7650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54183/0 (socket says 172.21.15.45:54183) 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26ead9d640 1 -- 172.21.15.45:0/2915652931 learned_addr learned my addr 172.21.15.45:0/2915652931 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:18.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26d37fe640 10 client.?.objecter ms_handle_connect 0x7f26ec0a9d70 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.319+0000 7f26d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88132398 0 0) 0x7f26ec1e8470 con 0x7f26ec0a9d70 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d8004eb0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705597787 0 0) 0x7f26d8004eb0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26ec1e8470 con 0x7f26ec0a9d70 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f26dc001910 con 0x7f26ec0a9d70 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): dump: 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:18.301 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:18.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:18.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1330440562 0 0) 0x7f26ec1e8470 con 0x7f26ec0a9d70 2024-03-29T10:59:18.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 >> v1:172.21.15.116:6789/0 conn(0x7f26ec1457f0 legacy=0x7f26ec1ee980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.320+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 >> v1:172.21.15.45:6789/0 conn(0x7f26ec13f650 legacy=0x7f26ec1e7d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 1 monclient: found mon.c 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec144290 con 0x7f26ec0a9d70 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26f124a640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26f124a640 10 monclient: _renew_subs 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26f124a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec144480 con 0x7f26ec0a9d70 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.322+0000 7f26f124a640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.323+0000 7f26f124a640 10 monclient: _renew_subs 2024-03-29T10:59:18.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.323+0000 7f26f124a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.323+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec138ea0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.323+0000 7f26f124a640 1 librados: init done 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26dc0047e0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f26dc004af0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26d37fe640 10 monclient: dump: 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:18.306 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:18.307 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:18.307 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:18.307 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:18.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.324+0000 7f26e959a640 4 set_mon_vals no callback set 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.325+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f26dc01ab40 con 0x7f26ec0a9d70 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26ea59c640 10 monclient: get_auth_request con 0x7f26d806dac0 auth_method 0 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f26dc084b40 con 0x7f26ec0a9d70 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26d37fe640 10 client.4505.objecter ms_dispatch 0x7f26ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26d37fe640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26d37fe640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:18.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26d37fe640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T10:59:18.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.326+0000 7f26d37fe640 10 client.4505.objecter ms_handle_connect 0x7f26d806dac0 2024-03-29T10:59:18.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.327+0000 7f26f124a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.328+0000 7f26f124a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.328+0000 7f26f124a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:18.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.328+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26cc005080 con 0x7f26ec0a9d70 2024-03-29T10:59:18.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.339+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26dc050480 con 0x7f26ec0a9d70 2024-03-29T10:59:18.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.339+0000 7f26d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.339+0000 7f26d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:18.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.672+0000 7f26f124a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.672+0000 7f26f124a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.672+0000 7f26f124a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:18.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.672+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26cc005330 con 0x7f26ec0a9d70 2024-03-29T10:59:18.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.673+0000 7f26d37fe640 1 -- 172.21.15.45:0/2915652931 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f26dc0841f0 con 0x7f26ec0a9d70 2024-03-29T10:59:18.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.673+0000 7f26d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.673+0000 7f26d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:18.653 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:18.653 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:18.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.676+0000 7f26f124a640 10 librados: watch_flush enter 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.676+0000 7f26f124a640 10 librados: watch_flush exit 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.676+0000 7f26f124a640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.676+0000 7f26f124a640 10 client.4505.objecter successfully canceled tick 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.680+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 >> v1:172.21.15.116:6816/3290579615 conn(0x7f26d806dac0 legacy=0x7f26d806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.680+0000 7f26f124a640 10 monclient: shutdown 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.680+0000 7f26f124a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.680+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 >> v1:172.21.15.45:6790/0 conn(0x7f26ec0a9d70 legacy=0x7f26ec1e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.680+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 shutdown_connections 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.680+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 >> 172.21.15.45:0/2915652931 conn(0x7f26ec066a00 msgr2=0x7f26ec1daae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.681+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 shutdown_connections 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.681+0000 7f26f124a640 1 -- 172.21.15.45:0/2915652931 wait complete. 2024-03-29T10:59:18.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:18.681+0000 7f26f124a640 1 librados: shutdown 2024-03-29T10:59:18.678 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:25.485 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.659+0000 7fa25783e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.659+0000 7fa25783e640 10 monclient: build_initial_monmap 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.659+0000 7fa25783e640 10 monclient: monmap: 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:25.660754+0000 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:25.660754+0000 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.640 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:25.641 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:25.641 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:25.641 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.661+0000 7fa25783e640 1 Processor -- start 2024-03-29T10:59:25.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.661+0000 7fa25783e640 1 -- start start 2024-03-29T10:59:25.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.661+0000 7fa25783e640 10 monclient: init 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient(hunting): picked mon.noname-a con 0x7fa2501457f0 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient(hunting): picked mon.noname-b con 0x7fa25013f650 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient(hunting): picked mon.noname-c con 0x7fa2500a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500aaef0 con 0x7fa2501457f0 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa250145bd0 con 0x7fa2500a9d70 2024-03-29T10:59:25.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa250145eb0 con 0x7fa25013f650 2024-03-29T10:59:25.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:25.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa25783e640 10 monclient(hunting): authenticate will time out at 1239.931541s 2024-03-29T10:59:25.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa255ddb640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa2500a9d70 0x7fa25014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:34588/0 (socket says 172.21.15.45:34588) 2024-03-29T10:59:25.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.662+0000 7fa255ddb640 1 -- 172.21.15.45:0/3908922534 learned_addr learned my addr 172.21.15.45:0/3908922534 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:25.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.663+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70769893 0 0) 0x7fa250145bd0 con 0x7fa2500a9d70 2024-03-29T10:59:25.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.663+0000 7fa247fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.663+0000 7fa247fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.663+0000 7fa247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.663+0000 7fa247fff640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:59:25.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.663+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa23c003520 con 0x7fa2500a9d70 2024-03-29T10:59:25.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714406944 0 0) 0x7fa23c003520 con 0x7fa2500a9d70 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 >> v1:172.21.15.45:6790/0 conn(0x7fa25013f650 legacy=0x7fa2500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 >> v1:172.21.15.45:6789/0 conn(0x7fa2501457f0 legacy=0x7fa25013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 1 monclient: found mon.noname-c 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa250146340 con 0x7fa2500a9d70 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa248001a10 con 0x7fa2500a9d70 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:25.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa247fff640 10 monclient: dump: 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa25783e640 5 monclient: authenticate success, global_id 4486 2024-03-29T10:59:25.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.664+0000 7fa25783e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa248001bf0 con 0x7fa2500a9d70 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa247fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa25783e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa25783e640 4 set_mon_vals no callback set 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa247fff640 1 -- 172.21.15.45:0/3908922534 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa248002040 con 0x7fa2500a9d70 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa247fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.665+0000 7fa247fff640 10 monclient: dump: 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 10 monclient: shutdown 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 1 -- 172.21.15.45:0/3908922534 >> v1:172.21.15.116:6789/0 conn(0x7fa2500a9d70 legacy=0x7fa25014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 1 -- 172.21.15.45:0/3908922534 shutdown_connections 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 1 -- 172.21.15.45:0/3908922534 >> 172.21.15.45:0/3908922534 conn(0x7fa250066a00 msgr2=0x7fa250066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 1 -- 172.21.15.45:0/3908922534 shutdown_connections 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.666+0000 7fa25783e640 1 -- 172.21.15.45:0/3908922534 wait complete. 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.667+0000 7fa25783e640 10 monclient: build_initial_monmap 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.667+0000 7fa25783e640 10 monclient: monmap: 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:25.650 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 1 librados: starting msgr at 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 1 librados: starting objecter 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 1 Processor -- start 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 1 -- start start 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 1 librados: setting wanted keys 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 1 librados: calling monclient init 2024-03-29T10:59:25.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.668+0000 7fa25783e640 10 monclient: init 2024-03-29T10:59:25.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:25.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:25.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient(hunting): picked mon.noname-a con 0x7fa2500a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:25.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient(hunting): picked mon.noname-b con 0x7fa25013f650 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:25.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient(hunting): picked mon.noname-c con 0x7fa2501457f0 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:25.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500ade40 con 0x7fa2500a9d70 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500ae360 con 0x7fa25013f650 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa2555da640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa2500a9d70 0x7fa250139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43604/0 (socket says 172.21.15.45:43604) 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa2555da640 1 -- 172.21.15.45:0/3015960171 learned_addr learned my addr 172.21.15.45:0/3015960171 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500ad890 con 0x7fa2501457f0 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa25783e640 10 monclient(hunting): authenticate will time out at 1239.938577s 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa245ffb640 10 client.?.objecter ms_handle_connect 0x7fa2500a9d70 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.669+0000 7fa245ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 client.?.objecter ms_handle_connect 0x7fa25013f650 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323550460 0 0) 0x7fa2500ade40 con 0x7fa2500a9d70 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa234004eb0 con 0x7fa2500a9d70 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 client.?.objecter ms_handle_connect 0x7fa2501457f0 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896862633 0 0) 0x7fa2500ae360 con 0x7fa25013f650 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.670+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2500ade40 con 0x7fa25013f650 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282452679 0 0) 0x7fa234004eb0 con 0x7fa2500a9d70 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2500ae360 con 0x7fa2500a9d70 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa240001930 con 0x7fa2500a9d70 2024-03-29T10:59:25.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603045057 0 0) 0x7fa2500ade40 con 0x7fa25013f650 2024-03-29T10:59:25.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa234004eb0 con 0x7fa25013f650 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa24c0018b0 con 0x7fa25013f650 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2161618720 0 0) 0x7fa2500ae360 con 0x7fa2500a9d70 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 >> v1:172.21.15.45:6790/0 conn(0x7fa2501457f0 legacy=0x7fa2501e2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 >> v1:172.21.15.116:6789/0 conn(0x7fa25013f650 legacy=0x7fa2501da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 monclient: found mon.a 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500ada20 con 0x7fa2500a9d70 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.671+0000 7fa245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:25.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 10 monclient: _renew_subs 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2500adbb0 con 0x7fa2500a9d70 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 10 monclient: _renew_subs 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25013a6e0 con 0x7fa2500a9d70 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.672+0000 7fa25783e640 1 librados: init done 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa240003610 con 0x7fa2500a9d70 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa245ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa2400039a0 con 0x7fa2500a9d70 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa245ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa245ffb640 10 monclient: dump: 2024-03-29T10:59:25.664 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.673+0000 7fa246ffd640 4 set_mon_vals no callback set 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.675+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fa240003b00 con 0x7fa2500a9d70 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.676+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fa240084920 con 0x7fa2500a9d70 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa245ffb640 10 client.4473.objecter ms_dispatch 0x7fa250000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa245ffb640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa245ffb640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:25.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa245ffb640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T10:59:25.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa25783e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa25783e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa25783e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:25.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa230005080 con 0x7fa2500a9d70 2024-03-29T10:59:25.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa254dd9640 10 monclient: get_auth_request con 0x7fa23406dec0 auth_method 0 2024-03-29T10:59:25.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.677+0000 7fa245ffb640 10 client.4473.objecter ms_handle_connect 0x7fa23406dec0 2024-03-29T10:59:25.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.687+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa240050200 con 0x7fa2500a9d70 2024-03-29T10:59:25.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.687+0000 7fa245ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:25.687+0000 7fa245ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:26.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.088+0000 7fa25783e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.088+0000 7fa25783e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.088+0000 7fa25783e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:26.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.088+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa230005330 con 0x7fa2500a9d70 2024-03-29T10:59:26.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.090+0000 7fa245ffb640 1 -- 172.21.15.45:0/3015960171 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fa240052130 con 0x7fa2500a9d70 2024-03-29T10:59:26.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.090+0000 7fa245ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.090+0000 7fa245ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:26.070 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:26.071 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:26.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 10 librados: watch_flush enter 2024-03-29T10:59:26.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 10 librados: watch_flush exit 2024-03-29T10:59:26.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T10:59:26.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 10 client.4473.objecter successfully canceled tick 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 >> v1:172.21.15.116:6816/3290579615 conn(0x7fa23406dec0 legacy=0x7fa234070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 10 monclient: shutdown 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 >> v1:172.21.15.45:6789/0 conn(0x7fa2500a9d70 legacy=0x7fa250139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 shutdown_connections 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.094+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 >> 172.21.15.45:0/3015960171 conn(0x7fa250066a00 msgr2=0x7fa250066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.095+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 shutdown_connections 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.095+0000 7fa25783e640 1 -- 172.21.15.45:0/3015960171 wait complete. 2024-03-29T10:59:26.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:26.095+0000 7fa25783e640 1 librados: shutdown 2024-03-29T10:59:26.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:32.898 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:33.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.076+0000 7f91c2045640 10 monclient: get_monmap_and_config 2024-03-29T10:59:33.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.076+0000 7f91c2045640 10 monclient: build_initial_monmap 2024-03-29T10:59:33.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.076+0000 7f91c2045640 10 monclient: monmap: 2024-03-29T10:59:33.056 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:33.056 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:33.077203+0000 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:33.077203+0000 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:33.057 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.077+0000 7f91c2045640 1 Processor -- start 2024-03-29T10:59:33.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.077+0000 7f91c2045640 1 -- start start 2024-03-29T10:59:33.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.077+0000 7f91c2045640 10 monclient: init 2024-03-29T10:59:33.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:33.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient(hunting): picked mon.noname-b con 0x7f91bc0b4710 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient(hunting): picked mon.noname-a con 0x7f91bc0b3970 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient(hunting): picked mon.noname-c con 0x7f91bc13f390 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc0af020 con 0x7f91bc0b3970 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc0b0910 con 0x7f91bc13f390 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc0b0bf0 con 0x7f91bc0b4710 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91c2045640 10 monclient(hunting): authenticate will time out at 1247.347718s 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91baffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f91bc0b3970 0x7f91bc0af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55724/0 (socket says 172.21.15.45:55724) 2024-03-29T10:59:33.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.078+0000 7f91baffd640 1 -- 172.21.15.45:0/1247412742 learned_addr learned my addr 172.21.15.45:0/1247412742 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159779097 0 0) 0x7f91bc0b0bf0 con 0x7f91bc0b4710 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4003520 con 0x7f91bc0b4710 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839914711 0 0) 0x7f91bc0af020 con 0x7f91bc0b3970 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4003840 con 0x7f91bc0b3970 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 299709791 0 0) 0x7f91a4003520 con 0x7f91bc0b4710 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 >> v1:172.21.15.116:6789/0 conn(0x7f91bc13f390 legacy=0x7f91bc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 >> v1:172.21.15.45:6789/0 conn(0x7f91bc0b3970 legacy=0x7f91bc0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 monclient: found mon.noname-b 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc0aeb00 con 0x7f91bc0b4710 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f91ac001670 con 0x7f91bc0b4710 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91ba7fc640 10 monclient: dump: 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91c2045640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:59:33.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.079+0000 7f91c2045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91ac001850 con 0x7f91bc0b4710 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91c2045640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91c2045640 4 set_mon_vals no callback set 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91ba7fc640 1 -- 172.21.15.45:0/1247412742 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f91ac001c60 con 0x7f91bc0b4710 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:33.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.082+0000 7f91ba7fc640 10 monclient: dump: 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 10 monclient: shutdown 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 1 -- 172.21.15.45:0/1247412742 >> v1:172.21.15.45:6790/0 conn(0x7f91bc0b4710 legacy=0x7f91bc0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 1 -- 172.21.15.45:0/1247412742 shutdown_connections 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 1 -- 172.21.15.45:0/1247412742 >> 172.21.15.45:0/1247412742 conn(0x7f91bc066a00 msgr2=0x7f91bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:33.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 1 -- 172.21.15.45:0/1247412742 shutdown_connections 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.083+0000 7f91c2045640 1 -- 172.21.15.45:0/1247412742 wait complete. 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.084+0000 7f91c2045640 10 monclient: build_initial_monmap 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.084+0000 7f91c2045640 10 monclient: monmap: 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:33.066 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.084+0000 7f91c2045640 1 librados: starting msgr at 2024-03-29T10:59:33.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.084+0000 7f91c2045640 1 librados: starting objecter 2024-03-29T10:59:33.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.085+0000 7f91c2045640 1 Processor -- start 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.085+0000 7f91c2045640 1 -- start start 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.085+0000 7f91c2045640 1 librados: setting wanted keys 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.085+0000 7f91c2045640 1 librados: calling monclient init 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.085+0000 7f91c2045640 10 monclient: init 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient(hunting): picked mon.noname-b con 0x7f91bc0b3970 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient(hunting): picked mon.noname-a con 0x7f91bc0b4710 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient(hunting): picked mon.noname-c con 0x7f91bc13f390 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91baffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f91bc0b4710 0x7f91bc132c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56098/0 (socket says 172.21.15.45:56098) 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91baffd640 1 -- 172.21.15.45:0/109678967 learned_addr learned my addr 172.21.15.45:0/109678967 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc14ddd0 con 0x7f91bc0b4710 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc14e060 con 0x7f91bc0b3970 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 client.?.objecter ms_handle_connect 0x7f91bc0b4710 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 client.?.objecter ms_handle_connect 0x7f91bc0b3970 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 client.?.objecter ms_handle_connect 0x7f91bc13f390 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91bc14e720 con 0x7f91bc13f390 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91c2045640 10 monclient(hunting): authenticate will time out at 1247.355783s 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204668717 0 0) 0x7f91bc14ddd0 con 0x7f91bc0b4710 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a8004eb0 con 0x7f91bc0b4710 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614756137 0 0) 0x7f91bc14e720 con 0x7f91bc13f390 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.086+0000 7f91a3fff640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc14ddd0 con 0x7f91bc13f390 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788323797 0 0) 0x7f91bc14e060 con 0x7f91bc0b3970 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc14e720 con 0x7f91bc0b3970 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2181520436 0 0) 0x7f91bc14e720 con 0x7f91bc0b3970 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a8005670 con 0x7f91bc0b3970 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444927467 0 0) 0x7f91bc14ddd0 con 0x7f91bc13f390 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a8005c30 con 0x7f91bc13f390 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f91b4001a70 con 0x7f91bc13f390 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): dump: 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f91ac001c60 con 0x7f91bc0b3970 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): dump: 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3299465160 0 0) 0x7f91a8004eb0 con 0x7f91bc0b4710 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91bc14ddd0 con 0x7f91bc0b4710 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f91b00019f0 con 0x7f91bc0b4710 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient(hunting): dump: 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3845374708 0 0) 0x7f91a8005670 con 0x7f91bc0b3970 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 >> v1:172.21.15.45:6790/0 conn(0x7f91bc13f390 legacy=0x7f91bc1db770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 >> v1:172.21.15.45:6789/0 conn(0x7f91bc0b4710 legacy=0x7f91bc132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 monclient: found mon.b 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc133390 con 0x7f91bc0b3970 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.087+0000 7f91a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 5 monclient: authenticate success, global_id 4492 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 10 monclient: _renew_subs 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91bc133640 con 0x7f91bc0b3970 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 10 monclient: _renew_subs 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:33.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91bc142100 con 0x7f91bc0b3970 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.088+0000 7f91c2045640 1 librados: init done 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91ac001110 con 0x7f91bc0b3970 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f91ac001c60 con 0x7f91bc0b3970 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91a3fff640 10 monclient: dump: 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:33.082 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.089+0000 7f91b97fa640 4 set_mon_vals no callback set 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.090+0000 7f91c2045640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.090+0000 7f91c2045640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.090+0000 7f91c2045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.090+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c005080 con 0x7f91bc0b3970 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f91ac019710 con 0x7f91bc0b3970 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f91ac083550 con 0x7f91bc0b3970 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91a3fff640 10 client.4492.objecter ms_dispatch 0x7f91bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91a3fff640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91a3fff640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91a3fff640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T10:59:33.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.091+0000 7f91baffd640 10 monclient: get_auth_request con 0x7f91a806e620 auth_method 0 2024-03-29T10:59:33.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.092+0000 7f91a3fff640 10 client.4492.objecter ms_handle_connect 0x7f91a806e620 2024-03-29T10:59:33.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.101+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91ac04ee30 con 0x7f91bc0b3970 2024-03-29T10:59:33.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.101+0000 7f91a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.101+0000 7f91a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:33.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.495+0000 7f91c2045640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.495+0000 7f91c2045640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.495+0000 7f91c2045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:33.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.495+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f919c005330 con 0x7f91bc0b3970 2024-03-29T10:59:33.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.496+0000 7f91a3fff640 1 -- 172.21.15.45:0/109678967 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f91ac050d60 con 0x7f91bc0b3970 2024-03-29T10:59:33.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.496+0000 7f91a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.496+0000 7f91a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:33.476 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:33.477 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.499+0000 7f91c2045640 10 librados: watch_flush enter 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 10 librados: watch_flush exit 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 10 client.4492.objecter successfully canceled tick 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 >> v1:172.21.15.116:6816/3290579615 conn(0x7f91a806e620 legacy=0x7f91a8070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 10 monclient: shutdown 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:33.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 >> v1:172.21.15.116:6789/0 conn(0x7f91bc0b3970 legacy=0x7f91bc14d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 shutdown_connections 2024-03-29T10:59:33.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.500+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 >> 172.21.15.45:0/109678967 conn(0x7f91bc066a00 msgr2=0x7f91bc0b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:33.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.501+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 shutdown_connections 2024-03-29T10:59:33.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.501+0000 7f91c2045640 1 -- 172.21.15.45:0/109678967 wait complete. 2024-03-29T10:59:33.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:33.501+0000 7f91c2045640 1 librados: shutdown 2024-03-29T10:59:33.496 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:40.302 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:40.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.477+0000 7f58ec5f4640 10 monclient: get_monmap_and_config 2024-03-29T10:59:40.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.477+0000 7f58ec5f4640 10 monclient: build_initial_monmap 2024-03-29T10:59:40.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.477+0000 7f58ec5f4640 10 monclient: monmap: 2024-03-29T10:59:40.459 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:40.459 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:40.459 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:40.479022+0000 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:40.479022+0000 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.478+0000 7f58ec5f4640 1 Processor -- start 2024-03-29T10:59:40.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.479+0000 7f58ec5f4640 1 -- start start 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.479+0000 7f58ec5f4640 10 monclient: init 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f58e40a9ac0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f58e413ee90 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f58e40af900 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40afd10 con 0x7f58e40af900 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40af6f0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40aac80 con 0x7f58e413ee90 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:40.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.480+0000 7f58ec5f4640 10 monclient(hunting): authenticate will time out at 1254.750196s 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.487+0000 7f58eab91640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f58e40af900 0x7f58e4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55175/0 (socket says 172.21.15.45:55175) 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.487+0000 7f58eab91640 1 -- 172.21.15.45:0/1672564075 learned_addr learned my addr 172.21.15.45:0/1672564075 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342954839 0 0) 0x7f58e40afd10 con 0x7f58e40af900 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:59:40.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0003520 con 0x7f58e40af900 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71825324 0 0) 0x7f58d0003520 con 0x7f58e40af900 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 >> v1:172.21.15.45:6790/0 conn(0x7f58e413ee90 legacy=0x7f58e413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 >> v1:172.21.15.116:6789/0 conn(0x7f58e40a9ac0 legacy=0x7f58e40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 10 monclient: _finish_hunting 0 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.488+0000 7f58e938e640 1 monclient: found mon.noname-a 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:40.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40aaf60 con 0x7f58e40af900 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 10 monclient: _finish_auth 0 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 10 monclient: _check_auth_tickets 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f58d4004a10 con 0x7f58e40af900 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.489+0000 7f58e938e640 10 monclient: dump: 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:40.471 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.490+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d4004ca0 con 0x7f58e40af900 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.490+0000 7f58e938e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.490+0000 7f58e938e640 1 -- 172.21.15.45:0/1672564075 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f58d4004a10 con 0x7f58e40af900 2024-03-29T10:59:40.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.490+0000 7f58e938e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.490+0000 7f58e938e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.490+0000 7f58e938e640 10 monclient: dump: 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:40.473 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.491+0000 7f58ec5f4640 5 monclient: authenticate success, global_id 4482 2024-03-29T10:59:40.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.491+0000 7f58ec5f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:40.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.491+0000 7f58ec5f4640 4 set_mon_vals no callback set 2024-03-29T10:59:40.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.492+0000 7f58ec5f4640 10 monclient: shutdown 2024-03-29T10:59:40.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.492+0000 7f58ec5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:40.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.492+0000 7f58ec5f4640 1 -- 172.21.15.45:0/1672564075 >> v1:172.21.15.45:6789/0 conn(0x7f58e40af900 legacy=0x7f58e4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.492+0000 7f58ec5f4640 1 -- 172.21.15.45:0/1672564075 shutdown_connections 2024-03-29T10:59:40.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.492+0000 7f58ec5f4640 1 -- 172.21.15.45:0/1672564075 >> 172.21.15.45:0/1672564075 conn(0x7f58e40b35e0 msgr2=0x7f58e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:40.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.492+0000 7f58ec5f4640 1 -- 172.21.15.45:0/1672564075 shutdown_connections 2024-03-29T10:59:40.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.493+0000 7f58ec5f4640 1 -- 172.21.15.45:0/1672564075 wait complete. 2024-03-29T10:59:40.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.493+0000 7f58ec5f4640 10 monclient: build_initial_monmap 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.493+0000 7f58ec5f4640 10 monclient: monmap: 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 1 librados: starting msgr at 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 1 librados: starting objecter 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 1 Processor -- start 2024-03-29T10:59:40.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 1 -- start start 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 1 librados: setting wanted keys 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 1 librados: calling monclient init 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.494+0000 7f58ec5f4640 10 monclient: init 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f58e40a9ac0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f58e40af900 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f58e413ee90 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e41db720 con 0x7f58e40af900 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e41409b0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4140c90 con 0x7f58e413ee90 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:40.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.495+0000 7f58ec5f4640 10 monclient(hunting): authenticate will time out at 1254.765160s 2024-03-29T10:59:40.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.499+0000 7f58eab91640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f58e413ee90 0x7f58e41db010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58647/0 (socket says 172.21.15.45:58647) 2024-03-29T10:59:40.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.499+0000 7f58eab91640 1 -- 172.21.15.45:0/485673827 learned_addr learned my addr 172.21.15.45:0/485673827 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:40.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.499+0000 7f58daffd640 10 client.?.objecter ms_handle_connect 0x7f58e413ee90 2024-03-29T10:59:40.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.499+0000 7f58daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 client.?.objecter ms_handle_connect 0x7f58e40a9ac0 2024-03-29T10:59:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022138847 0 0) 0x7f58e41409b0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c8004eb0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418474194 0 0) 0x7f58e4140c90 con 0x7f58e413ee90 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e41409b0 con 0x7f58e413ee90 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 client.?.objecter ms_handle_connect 0x7f58e40af900 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.502+0000 7f58daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883076046 0 0) 0x7f58e41409b0 con 0x7f58e413ee90 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e4140c90 con 0x7f58e413ee90 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f58d4004a10 con 0x7f58e413ee90 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): dump: 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:40.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244554670 0 0) 0x7f58c8004eb0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e41409b0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f58dc001890 con 0x7f58e40a9ac0 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): dump: 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:40.490 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041438176 0 0) 0x7f58e41db720 con 0x7f58e40af900 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.503+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c8004eb0 con 0x7f58e40af900 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 287406604 0 0) 0x7f58e41409b0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 >> v1:172.21.15.45:6790/0 conn(0x7f58e413ee90 legacy=0x7f58e41db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 >> v1:172.21.15.45:6789/0 conn(0x7f58e40af900 legacy=0x7f58e41da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 1 monclient: found mon.b 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e41395a0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 5 monclient: authenticate success, global_id 4495 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 10 monclient: _renew_subs 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e41dc6c0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 10 monclient: _renew_subs 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e413a320 con 0x7f58e40a9ac0 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.504+0000 7f58ec5f4640 1 librados: init done 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.505+0000 7f58ec5f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.506+0000 7f58ec5f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.506+0000 7f58ec5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:40.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.506+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e4075f10 con 0x7f58e40a9ac0 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58dc001da0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f58dc003c90 con 0x7f58e40a9ac0 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58daffd640 10 monclient: dump: 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:40.494 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.508+0000 7f58e8b8d640 4 set_mon_vals no callback set 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.509+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f58dc019c60 con 0x7f58e40a9ac0 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.511+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f58dc083a90 con 0x7f58e40a9ac0 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.511+0000 7f58daffd640 10 client.4495.objecter ms_dispatch 0x7f58e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.511+0000 7f58daffd640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.511+0000 7f58daffd640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.511+0000 7f58daffd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T10:59:40.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.511+0000 7f58e9b8f640 10 monclient: get_auth_request con 0x7f58c806dfa0 auth_method 0 2024-03-29T10:59:40.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.515+0000 7f58daffd640 10 client.4495.objecter ms_handle_connect 0x7f58c806dfa0 2024-03-29T10:59:40.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.519+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58dc04f3d0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.519+0000 7f58daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.519+0000 7f58daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:40.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.844+0000 7f58ec5f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:40.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.844+0000 7f58ec5f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:40.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.844+0000 7f58ec5f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T10:59:40.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.844+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e40b3280 con 0x7f58e40a9ac0 2024-03-29T10:59:40.832 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:40.832 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:40.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.851+0000 7f58daffd640 1 -- 172.21.15.45:0/485673827 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f58dc04f3d0 con 0x7f58e40a9ac0 2024-03-29T10:59:40.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.851+0000 7f58daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:40.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.851+0000 7f58daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:40.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.854+0000 7f58ec5f4640 10 librados: watch_flush enter 2024-03-29T10:59:40.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.854+0000 7f58ec5f4640 10 librados: watch_flush exit 2024-03-29T10:59:40.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.854+0000 7f58ec5f4640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T10:59:40.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.855+0000 7f58ec5f4640 10 client.4495.objecter successfully canceled tick 2024-03-29T10:59:40.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.855+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 >> v1:172.21.15.116:6816/3290579615 conn(0x7f58c806dfa0 legacy=0x7f58c8070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.855+0000 7f58ec5f4640 10 monclient: shutdown 2024-03-29T10:59:40.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.855+0000 7f58ec5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:40.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.855+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 >> v1:172.21.15.116:6789/0 conn(0x7f58e40a9ac0 legacy=0x7f58e41dbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.856+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 shutdown_connections 2024-03-29T10:59:40.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.856+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 >> 172.21.15.45:0/485673827 conn(0x7f58e40b35e0 msgr2=0x7f58e40ad180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:40.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.856+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 shutdown_connections 2024-03-29T10:59:40.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.856+0000 7f58ec5f4640 1 -- 172.21.15.45:0/485673827 wait complete. 2024-03-29T10:59:40.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:40.856+0000 7f58ec5f4640 1 librados: shutdown 2024-03-29T10:59:40.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:47.662 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.829+0000 7fe61ea6e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.829+0000 7fe61ea6e640 10 monclient: build_initial_monmap 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.829+0000 7fe61ea6e640 10 monclient: monmap: 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:47.831443+0000 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:47.831443+0000 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:47.812 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:47.813 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:47.813 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:47.813 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:47.813 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:47.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.833+0000 7fe61ea6e640 1 Processor -- start 2024-03-29T10:59:47.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.833+0000 7fe61ea6e640 1 -- start start 2024-03-29T10:59:47.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.833+0000 7fe61ea6e640 10 monclient: init 2024-03-29T10:59:47.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.834+0000 7fe61ea6e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:47.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.834+0000 7fe61ea6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:47.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.834+0000 7fe61ea6e640 10 monclient(hunting): picked mon.noname-a con 0x7fe6180a9ac0 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:47.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.834+0000 7fe61ea6e640 10 monclient(hunting): picked mon.noname-b con 0x7fe61813ee90 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:47.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.836+0000 7fe61ea6e640 10 monclient(hunting): picked mon.noname-c con 0x7fe6180af900 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:47.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61ea6e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180afd10 con 0x7fe6180a9ac0 2024-03-29T10:59:47.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61ea6e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180af6f0 con 0x7fe6180af900 2024-03-29T10:59:47.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61ea6e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180aac80 con 0x7fe61813ee90 2024-03-29T10:59:47.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61ea6e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:47.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61ea6e640 10 monclient(hunting): authenticate will time out at 1262.107121s 2024-03-29T10:59:47.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61d00b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe6180af900 0x7fe618149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:50203/0 (socket says 172.21.15.45:50203) 2024-03-29T10:59:47.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.837+0000 7fe61d00b640 1 -- 172.21.15.45:0/822556697 learned_addr learned my addr 172.21.15.45:0/822556697 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:47.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226008908 0 0) 0x7fe6180afd10 con 0x7fe6180a9ac0 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe600005730 con 0x7fe6180a9ac0 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381825836 0 0) 0x7fe6180af6f0 con 0x7fe6180af900 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6180afd10 con 0x7fe6180af900 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3031474203 0 0) 0x7fe6180afd10 con 0x7fe6180af900 2024-03-29T10:59:47.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 >> v1:172.21.15.45:6790/0 conn(0x7fe61813ee90 legacy=0x7fe61813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 >> v1:172.21.15.45:6789/0 conn(0x7fe6180a9ac0 legacy=0x7fe6180adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180aaf60 con 0x7fe6180af900 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe61ea6e640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe61ea6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6100019f0 con 0x7fe6180af900 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 10 monclient: dump: 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe61ea6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.838+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe610001c40 con 0x7fe6180af900 2024-03-29T10:59:47.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe60f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe60f7fe640 1 -- 172.21.15.45:0/822556697 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe610001fd0 con 0x7fe6180af900 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe60f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe60f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe60f7fe640 10 monclient: dump: 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:47.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe61ea6e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:47.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.839+0000 7fe61ea6e640 4 set_mon_vals no callback set 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.841+0000 7fe61ea6e640 10 monclient: shutdown 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.841+0000 7fe61ea6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.841+0000 7fe61ea6e640 1 -- 172.21.15.45:0/822556697 >> v1:172.21.15.116:6789/0 conn(0x7fe6180af900 legacy=0x7fe618149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.842+0000 7fe61ea6e640 1 -- 172.21.15.45:0/822556697 shutdown_connections 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.842+0000 7fe61ea6e640 1 -- 172.21.15.45:0/822556697 >> 172.21.15.45:0/822556697 conn(0x7fe6180b35e0 msgr2=0x7fe6180b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.842+0000 7fe61ea6e640 1 -- 172.21.15.45:0/822556697 shutdown_connections 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.842+0000 7fe61ea6e640 1 -- 172.21.15.45:0/822556697 wait complete. 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 10 monclient: build_initial_monmap 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 10 monclient: monmap: 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:47.825 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 1 librados: starting msgr at 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 1 librados: starting objecter 2024-03-29T10:59:47.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 1 Processor -- start 2024-03-29T10:59:47.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 1 -- start start 2024-03-29T10:59:47.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 1 librados: setting wanted keys 2024-03-29T10:59:47.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 1 librados: calling monclient init 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.843+0000 7fe61ea6e640 10 monclient: init 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.844+0000 7fe61ea6e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.844+0000 7fe61ea6e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.844+0000 7fe61ea6e640 10 monclient(hunting): picked mon.noname-b con 0x7fe6180a9ac0 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.844+0000 7fe61ea6e640 10 monclient(hunting): picked mon.noname-c con 0x7fe6180af900 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe60ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6180af900 0x7fe6181db8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50406/0 (socket says 172.21.15.45:50406) 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe61ea6e640 10 monclient(hunting): picked mon.noname-a con 0x7fe61813ee90 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe60ffff640 1 -- 172.21.15.45:0/2753805707 learned_addr learned my addr 172.21.15.45:0/2753805707 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6181db3d0 con 0x7fe61813ee90 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61811b920 con 0x7fe6180a9ac0 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618072080 con 0x7fe6180af900 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe61ea6e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe61ea6e640 10 monclient(hunting): authenticate will time out at 1262.116604s 2024-03-29T10:59:47.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe60d7fa640 10 client.?.objecter ms_handle_connect 0x7fe6180af900 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe60d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.846+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320731462 0 0) 0x7fe618072080 con 0x7fe6180af900 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.847+0000 7fe60d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.847+0000 7fe60d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.847+0000 7fe60d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.847+0000 7fe60d7fa640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.847+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604004eb0 con 0x7fe6180af900 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 client.?.objecter ms_handle_connect 0x7fe61813ee90 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 994568618 0 0) 0x7fe604004eb0 con 0x7fe6180af900 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe618072080 con 0x7fe6180af900 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86259295 0 0) 0x7fe6181db3d0 con 0x7fe61813ee90 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604004eb0 con 0x7fe61813ee90 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6080018c0 con 0x7fe6180af900 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): dump: 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:47.839 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.850+0000 7fe60d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 10 client.?.objecter ms_handle_connect 0x7fe6180a9ac0 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007967995 0 0) 0x7fe61811b920 con 0x7fe6180a9ac0 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.851+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6181db3d0 con 0x7fe6180a9ac0 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 973023715 0 0) 0x7fe618072080 con 0x7fe6180af900 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 >> v1:172.21.15.116:6789/0 conn(0x7fe6180a9ac0 legacy=0x7fe618071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 >> v1:172.21.15.45:6789/0 conn(0x7fe61813ee90 legacy=0x7fe61813a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 1 monclient: found mon.c 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61806e7a0 con 0x7fe6180af900 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe60d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 10 monclient: _renew_subs 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181dbfc0 con 0x7fe6180af900 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 10 monclient: _renew_subs 2024-03-29T10:59:47.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:47.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.852+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61813abd0 con 0x7fe6180af900 2024-03-29T10:59:47.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.853+0000 7fe61ea6e640 1 librados: init done 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.856+0000 7fe61ea6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.856+0000 7fe61ea6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.856+0000 7fe61ea6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.856+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6181dc250 con 0x7fe6180af900 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe608003610 con 0x7fe6180af900 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6080039a0 con 0x7fe6180af900 2024-03-29T10:59:47.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60d7fa640 10 monclient: dump: 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.857+0000 7fe60e7fc640 4 set_mon_vals no callback set 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.858+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe608003bc0 con 0x7fe6180af900 2024-03-29T10:59:47.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.859+0000 7fe61c80a640 10 monclient: get_auth_request con 0x7fe60406dea0 auth_method 0 2024-03-29T10:59:47.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.859+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fe608084a30 con 0x7fe6180af900 2024-03-29T10:59:47.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.859+0000 7fe60d7fa640 10 client.4526.objecter ms_dispatch 0x7fe618000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:47.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.859+0000 7fe60d7fa640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:47.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.859+0000 7fe60d7fa640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:47.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.859+0000 7fe60d7fa640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T10:59:47.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.861+0000 7fe60d7fa640 10 client.4526.objecter ms_handle_connect 0x7fe60406dea0 2024-03-29T10:59:47.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.868+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe608050490 con 0x7fe6180af900 2024-03-29T10:59:47.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.868+0000 7fe60d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:47.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:47.868+0000 7fe60d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:48.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.196+0000 7fe61ea6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.196+0000 7fe61ea6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.196+0000 7fe61ea6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T10:59:48.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.196+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe61806e930 con 0x7fe6180af900 2024-03-29T10:59:48.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.197+0000 7fe60d7fa640 1 -- 172.21.15.45:0/2753805707 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fe61806e930 con 0x7fe6180af900 2024-03-29T10:59:48.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.197+0000 7fe60d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.197+0000 7fe60d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:48.178 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:48.179 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:48.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 10 librados: watch_flush enter 2024-03-29T10:59:48.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 10 librados: watch_flush exit 2024-03-29T10:59:48.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T10:59:48.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 10 client.4526.objecter successfully canceled tick 2024-03-29T10:59:48.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe60406dea0 legacy=0x7fe604070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 10 monclient: shutdown 2024-03-29T10:59:48.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:48.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 >> v1:172.21.15.45:6790/0 conn(0x7fe6180af900 legacy=0x7fe6181db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 shutdown_connections 2024-03-29T10:59:48.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 >> 172.21.15.45:0/2753805707 conn(0x7fe6180b35e0 msgr2=0x7fe6180adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:48.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 shutdown_connections 2024-03-29T10:59:48.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.201+0000 7fe61ea6e640 1 -- 172.21.15.45:0/2753805707 wait complete. 2024-03-29T10:59:48.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:48.202+0000 7fe61ea6e640 1 librados: shutdown 2024-03-29T10:59:48.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:55.008 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:55.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.181+0000 7f2dc3d0e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:55.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.181+0000 7f2dc3d0e640 10 monclient: build_initial_monmap 2024-03-29T10:59:55.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.181+0000 7f2dc3d0e640 10 monclient: monmap: 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:59:55.182140+0000 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:59:55.182140+0000 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:55.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.182+0000 7f2dc3d0e640 1 Processor -- start 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.182+0000 7f2dc3d0e640 1 -- start start 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.182+0000 7f2dc3d0e640 10 monclient: init 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dbc0b4100 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dbc13ee90 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dbc0af900 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0afd10 con 0x7f2dbc0b4100 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0af6f0 con 0x7f2dbc0af900 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0aac80 con 0x7f2dbc13ee90 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:55.165 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.183+0000 7f2dc3d0e640 10 monclient(hunting): authenticate will time out at 1269.452658s 2024-03-29T10:59:55.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.184+0000 7f2dc1aaa640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2dbc0b4100 0x7f2dbc0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59762/0 (socket says 172.21.15.45:59762) 2024-03-29T10:59:55.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.184+0000 7f2dc1aaa640 1 -- 172.21.15.45:0/209024041 learned_addr learned my addr 172.21.15.45:0/209024041 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:55.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.188+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456360963 0 0) 0x7f2dbc0afd10 con 0x7f2dbc0b4100 2024-03-29T10:59:55.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.188+0000 7f2dc0aa8640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.188+0000 7f2dc0aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.188+0000 7f2dc0aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.188+0000 7f2dc0aa8640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:59:55.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.188+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da8003520 con 0x7f2dbc0b4100 2024-03-29T10:59:55.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.189+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1212578471 0 0) 0x7f2da8003520 con 0x7f2dbc0b4100 2024-03-29T10:59:55.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.189+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 >> v1:172.21.15.45:6790/0 conn(0x7f2dbc13ee90 legacy=0x7f2dbc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:55.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.189+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 >> v1:172.21.15.116:6789/0 conn(0x7f2dbc0af900 legacy=0x7f2dbc14be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:55.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: _finish_hunting 0 2024-03-29T10:59:55.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 1 monclient: found mon.noname-a 2024-03-29T10:59:55.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:55.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T10:59:55.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0aaf60 con 0x7f2dbc0b4100 2024-03-29T10:59:55.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: _finish_auth 0 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: _check_auth_tickets 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2db8001690 con 0x7f2dbc0b4100 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: dump: 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:55.172 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc3d0e640 5 monclient: authenticate success, global_id 4494 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc3d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db80018f0 con 0x7f2dbc0b4100 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 1 -- 172.21.15.45:0/209024041 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2db8001d40 con 0x7f2dbc0b4100 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc0aa8640 10 monclient: dump: 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.173 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc3d0e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.191+0000 7f2dc3d0e640 4 set_mon_vals no callback set 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.192+0000 7f2dc3d0e640 10 monclient: shutdown 2024-03-29T10:59:55.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.192+0000 7f2dc3d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:55.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.192+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/209024041 >> v1:172.21.15.45:6789/0 conn(0x7f2dbc0b4100 legacy=0x7f2dbc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.196+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/209024041 shutdown_connections 2024-03-29T10:59:55.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.196+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/209024041 >> 172.21.15.45:0/209024041 conn(0x7f2dbc0b35e0 msgr2=0x7f2dbc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:55.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.197+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/209024041 shutdown_connections 2024-03-29T10:59:55.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.197+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/209024041 wait complete. 2024-03-29T10:59:55.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.198+0000 7f2dc3d0e640 10 monclient: build_initial_monmap 2024-03-29T10:59:55.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.198+0000 7f2dc3d0e640 10 monclient: monmap: 2024-03-29T10:59:55.179 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T10:59:55.179 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:55.179 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T10:59:55.179 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.198+0000 7f2dc3d0e640 1 librados: starting msgr at 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.198+0000 7f2dc3d0e640 1 librados: starting objecter 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.199+0000 7f2dc3d0e640 1 Processor -- start 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.199+0000 7f2dc3d0e640 1 -- start start 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.199+0000 7f2dc3d0e640 1 librados: setting wanted keys 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.199+0000 7f2dc3d0e640 1 librados: calling monclient init 2024-03-29T10:59:55.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.199+0000 7f2dc3d0e640 10 monclient: init 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dbc0af900 addr v1:172.21.15.116:6789/0 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dbc0b4100 addr v1:172.21.15.45:6790/0 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dbc13ee90 addr v1:172.21.15.45:6789/0 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc129980 con 0x7f2dbc13ee90 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.200+0000 7f2dc3d0e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc129c60 con 0x7f2dbc0af900 2024-03-29T10:59:55.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.201+0000 7f2dc3d0e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc1280a0 con 0x7f2dbc0b4100 2024-03-29T10:59:55.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.201+0000 7f2dc3d0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:55.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.201+0000 7f2dc3d0e640 10 monclient(hunting): authenticate will time out at 1269.470113s 2024-03-29T10:59:55.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.202+0000 7f2dc22ab640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2dbc13ee90 0x7f2dbc127990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41770/0 (socket says 172.21.15.45:41770) 2024-03-29T10:59:55.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.202+0000 7f2dc22ab640 1 -- 172.21.15.45:0/671477126 learned_addr learned my addr 172.21.15.45:0/671477126 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T10:59:55.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 10 client.?.objecter ms_handle_connect 0x7f2dbc13ee90 2024-03-29T10:59:55.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338322994 0 0) 0x7f2dbc129980 con 0x7f2dbc13ee90 2024-03-29T10:59:55.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:59:55.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.203+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da0004eb0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3823680806 0 0) 0x7f2da0004eb0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dbc129980 con 0x7f2dbc13ee90 2024-03-29T10:59:55.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2dac0017f0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:55.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 monclient(hunting): dump: 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 client.?.objecter ms_handle_connect 0x7f2dbc0b4100 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597050560 0 0) 0x7f2dbc1280a0 con 0x7f2dbc0b4100 2024-03-29T10:59:55.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.205+0000 7f2db27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da0004eb0 con 0x7f2dbc0b4100 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3770841353 0 0) 0x7f2dbc129980 con 0x7f2dbc13ee90 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 >> v1:172.21.15.45:6790/0 conn(0x7f2dbc0b4100 legacy=0x7f2dbc127280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.206+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 >> v1:172.21.15.116:6789/0 conn(0x7f2dbc0af900 legacy=0x7f2dbc13c2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 1 monclient: found mon.a 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:55.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc12a400 con 0x7f2dbc13ee90 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.208+0000 7f2db27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dac001dc0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2dac004aa0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db27fc640 10 monclient: dump: 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.190 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T10:59:55.191 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.191 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T10:59:55.191 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T10:59:55.191 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T10:59:55.191 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T10:59:55.191 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T10:59:55.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:59:55.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 10 monclient: _renew_subs 2024-03-29T10:59:55.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc12a6e0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2db37fe640 4 set_mon_vals no callback set 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 10 monclient: _renew_subs 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc12aca0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.209+0000 7f2dc3d0e640 1 librados: init done 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.211+0000 7f2dc3d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.211+0000 7f2dc3d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.211+0000 7f2dc3d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.211+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc06dfd0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.212+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f2dac01aa70 con 0x7f2dbc13ee90 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.212+0000 7f2dc1aaa640 10 monclient: get_auth_request con 0x7f2da0076420 auth_method 0 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.213+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f2dac084ac0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.213+0000 7f2db27fc640 10 client.4497.objecter ms_dispatch 0x7f2dbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:55.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.213+0000 7f2db27fc640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:55.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.213+0000 7f2db27fc640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:55.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.213+0000 7f2db27fc640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-29T10:59:55.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.213+0000 7f2db27fc640 10 client.4497.objecter ms_handle_connect 0x7f2da0076420 2024-03-29T10:59:55.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.220+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dac0501d0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.220+0000 7f2db27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.220+0000 7f2db27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:55.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.548+0000 7f2dc3d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.548+0000 7f2dc3d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.548+0000 7f2dc3d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T10:59:55.529 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.548+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dbc075f10 con 0x7f2dbc13ee90 2024-03-29T10:59:55.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.550+0000 7f2db27fc640 1 -- 172.21.15.45:0/671477126 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f2dac0522d0 con 0x7f2dbc13ee90 2024-03-29T10:59:55.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.550+0000 7f2db27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.550+0000 7f2db27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:55.531 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T10:59:55.531 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 10 librados: watch_flush enter 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 10 librados: watch_flush exit 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 10 client.4497.objecter successfully canceled tick 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 >> v1:172.21.15.116:6816/3290579615 conn(0x7f2da0076420 legacy=0x7f2da00788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 10 monclient: shutdown 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:55.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 >> v1:172.21.15.45:6789/0 conn(0x7f2dbc13ee90 legacy=0x7f2dbc127990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 shutdown_connections 2024-03-29T10:59:55.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.554+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 >> 172.21.15.45:0/671477126 conn(0x7f2dbc0b35e0 msgr2=0x7f2dbc0acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:55.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.556+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 shutdown_connections 2024-03-29T10:59:55.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.556+0000 7f2dc3d0e640 1 -- 172.21.15.45:0/671477126 wait complete. 2024-03-29T10:59:55.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T10:59:55.558+0000 7f2dc3d0e640 1 librados: shutdown 2024-03-29T10:59:55.555 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:02.362 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:02.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.518+0000 7f703edba640 10 monclient: get_monmap_and_config 2024-03-29T11:00:02.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.518+0000 7f703edba640 10 monclient: build_initial_monmap 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.518+0000 7f703edba640 10 monclient: monmap: 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:02.519368+0000 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:02.519368+0000 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:02.499 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.519+0000 7f703edba640 1 Processor -- start 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.519+0000 7f703edba640 1 -- start start 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.519+0000 7f703edba640 10 monclient: init 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient(hunting): picked mon.noname-b con 0x7f70380b0770 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:02.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient(hunting): picked mon.noname-a con 0x7f703814a700 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient(hunting): picked mon.noname-c con 0x7f70380ac5a0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380aba70 con 0x7f703814a700 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380aecb0 con 0x7f70380ac5a0 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380af150 con 0x7f70380b0770 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.520+0000 7f703edba640 10 monclient(hunting): authenticate will time out at 1276.789930s 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f703cb56640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f70380b0770 0x7f7038149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49598/0 (socket says 172.21.15.45:49598) 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f703cb56640 1 -- 172.21.15.45:0/1120445964 learned_addr learned my addr 172.21.15.45:0/1120445964 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873333631 0 0) 0x7f70380af150 con 0x7f70380b0770 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f70377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f70377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f70377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f70377fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:00:02.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.521+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7028003520 con 0x7f70380b0770 2024-03-29T11:00:02.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281215182 0 0) 0x7f70380aecb0 con 0x7f70380ac5a0 2024-03-29T11:00:02.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70380af150 con 0x7f70380ac5a0 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715333818 0 0) 0x7f7028003520 con 0x7f70380b0770 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 >> v1:172.21.15.116:6789/0 conn(0x7f70380ac5a0 legacy=0x7f7038151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.522+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 >> v1:172.21.15.45:6789/0 conn(0x7f703814a700 legacy=0x7f70380abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 1 monclient: found mon.noname-b 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380ac9d0 con 0x7f70380b0770 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:02.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f702c001690 con 0x7f70380b0770 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: dump: 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:02.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f702c0018b0 con 0x7f70380b0770 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 1 -- 172.21.15.45:0/1120445964 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f702c001c40 con 0x7f70380b0770 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f70377fe640 10 monclient: dump: 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:02.508 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f703edba640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f703edba640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.523+0000 7f703edba640 4 set_mon_vals no callback set 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.524+0000 7f703edba640 10 monclient: shutdown 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.524+0000 7f703edba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.524+0000 7f703edba640 1 -- 172.21.15.45:0/1120445964 >> v1:172.21.15.45:6790/0 conn(0x7f70380b0770 legacy=0x7f7038149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.524+0000 7f703edba640 1 -- 172.21.15.45:0/1120445964 shutdown_connections 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.524+0000 7f703edba640 1 -- 172.21.15.45:0/1120445964 >> 172.21.15.45:0/1120445964 conn(0x7f703806b660 msgr2=0x7f703806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.525+0000 7f703edba640 1 -- 172.21.15.45:0/1120445964 shutdown_connections 2024-03-29T11:00:02.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.525+0000 7f703edba640 1 -- 172.21.15.45:0/1120445964 wait complete. 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.526+0000 7f703edba640 10 monclient: build_initial_monmap 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.526+0000 7f703edba640 10 monclient: monmap: 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:02.512 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.526+0000 7f703edba640 1 librados: starting msgr at 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.526+0000 7f703edba640 1 librados: starting objecter 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.526+0000 7f703edba640 1 Processor -- start 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.527+0000 7f703edba640 1 -- start start 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.527+0000 7f703edba640 1 librados: setting wanted keys 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.527+0000 7f703edba640 1 librados: calling monclient init 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.527+0000 7f703edba640 10 monclient: init 2024-03-29T11:00:02.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient(hunting): picked mon.noname-c con 0x7f70380ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient(hunting): picked mon.noname-a con 0x7f70380b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient(hunting): picked mon.noname-b con 0x7f703814a700 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7038128730 con 0x7f70380b0770 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70381266c0 con 0x7f703814a700 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70381239d0 con 0x7f70380ac5a0 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703edba640 10 monclient(hunting): authenticate will time out at 1276.798432s 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703d357640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f703814a700 0x7f70381e6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:44500/0 (socket says 172.21.15.45:44500) 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f703d357640 1 -- 172.21.15.45:0/1734040252 learned_addr learned my addr 172.21.15.45:0/1734040252 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f70357fa640 10 client.?.objecter ms_handle_connect 0x7f703814a700 2024-03-29T11:00:02.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f70357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582294251 0 0) 0x7f70381266c0 con 0x7f703814a700 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f70357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f70357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.529+0000 7f70357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c004eb0 con 0x7f703814a700 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 client.?.objecter ms_handle_connect 0x7f70380ac5a0 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645706397 0 0) 0x7f70381239d0 con 0x7f70380ac5a0 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:00:02.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70381266c0 con 0x7f70380ac5a0 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4147544426 0 0) 0x7f70381266c0 con 0x7f70380ac5a0 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70381239d0 con 0x7f70380ac5a0 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f702c0015d0 con 0x7f70380ac5a0 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): dump: 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:02.516 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 647609797 0 0) 0x7f70381239d0 con 0x7f70380ac5a0 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 >> v1:172.21.15.116:6789/0 conn(0x7f703814a700 legacy=0x7f70381e6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.530+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 >> v1:172.21.15.45:6789/0 conn(0x7f70380b0770 legacy=0x7f70381451f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 1 monclient: found mon.c 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038123b60 con 0x7f70380ac5a0 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 10 monclient: _finish_auth 0 2024-03-29T11:00:02.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f70357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f703edba640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f703edba640 10 monclient: _renew_subs 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f703edba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.531+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70381272f0 con 0x7f70380ac5a0 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.532+0000 7f703edba640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.532+0000 7f703edba640 10 monclient: _renew_subs 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.532+0000 7f703edba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.532+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70381dfa90 con 0x7f70380ac5a0 2024-03-29T11:00:02.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.532+0000 7f703edba640 1 librados: init done 2024-03-29T11:00:02.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.533+0000 7f703edba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.533+0000 7f703edba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.533+0000 7f703edba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:02.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.533+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70380ab560 con 0x7f70380ac5a0 2024-03-29T11:00:02.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f702c004a90 con 0x7f70380ac5a0 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f702c004c00 con 0x7f70380ac5a0 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 10 monclient: dump: 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:02.520 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f7036ffd640 4 set_mon_vals no callback set 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.534+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f702c01abd0 con 0x7f70380ac5a0 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f702c0848c0 con 0x7f70380ac5a0 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f70357fa640 10 client.4535.objecter ms_dispatch 0x7f7038000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f70357fa640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f70357fa640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f70357fa640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f7037fff640 10 monclient: get_auth_request con 0x7f701c06dd00 auth_method 0 2024-03-29T11:00:02.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.535+0000 7f70357fa640 10 client.4535.objecter ms_handle_connect 0x7f701c06dd00 2024-03-29T11:00:02.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.542+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f702c050320 con 0x7f70380ac5a0 2024-03-29T11:00:02.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.542+0000 7f70357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.542+0000 7f70357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:02.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.868+0000 7f703edba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:02.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.868+0000 7f703edba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:02.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.868+0000 7f703edba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:02.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.868+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7038126900 con 0x7f70380ac5a0 2024-03-29T11:00:02.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.871+0000 7f70357fa640 1 -- 172.21.15.45:0/1734040252 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f7038126900 con 0x7f70380ac5a0 2024-03-29T11:00:02.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.871+0000 7f70357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:02.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.871+0000 7f70357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:02.851 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:02.851 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.874+0000 7f703edba640 10 librados: watch_flush enter 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.874+0000 7f703edba640 10 librados: watch_flush exit 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.874+0000 7f703edba640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.874+0000 7f703edba640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.875+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 >> v1:172.21.15.116:6816/3290579615 conn(0x7f701c06dd00 legacy=0x7f701c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.875+0000 7f703edba640 10 monclient: shutdown 2024-03-29T11:00:02.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.875+0000 7f703edba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:02.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.875+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 >> v1:172.21.15.45:6790/0 conn(0x7f70380ac5a0 legacy=0x7f7038126be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.875+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 shutdown_connections 2024-03-29T11:00:02.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.875+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 >> 172.21.15.45:0/1734040252 conn(0x7f703806b660 msgr2=0x7f70380b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:02.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.876+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 shutdown_connections 2024-03-29T11:00:02.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.876+0000 7f703edba640 1 -- 172.21.15.45:0/1734040252 wait complete. 2024-03-29T11:00:02.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:02.876+0000 7f703edba640 1 librados: shutdown 2024-03-29T11:00:02.873 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:09.680 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.845+0000 7f62ada2d640 10 monclient: get_monmap_and_config 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.845+0000 7f62ada2d640 10 monclient: build_initial_monmap 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.846+0000 7f62ada2d640 10 monclient: monmap: 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:09.846828+0000 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:09.846828+0000 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:09.826 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:09.827 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:09.827 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:09.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.847+0000 7f62ada2d640 1 Processor -- start 2024-03-29T11:00:09.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.847+0000 7f62ada2d640 1 -- start start 2024-03-29T11:00:09.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.847+0000 7f62ada2d640 10 monclient: init 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient(hunting): picked mon.noname-c con 0x7f62a806ec80 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient(hunting): picked mon.noname-b con 0x7f62a806fee0 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient(hunting): picked mon.noname-a con 0x7f62a813f3f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a80af000 con 0x7f62a813f3f0 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a806fc20 con 0x7f62a806ec80 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a8071dd0 con 0x7f62a806fee0 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:09.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.848+0000 7f62ada2d640 10 monclient(hunting): authenticate will time out at 1284.117166s 2024-03-29T11:00:09.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62aca2b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f62a806ec80 0x7f62a80ae8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:34270/0 (socket says 172.21.15.45:34270) 2024-03-29T11:00:09.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62aca2b640 1 -- 172.21.15.45:0/626765130 learned_addr learned my addr 172.21.15.45:0/626765130 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:09.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786217625 0 0) 0x7f62a806fc20 con 0x7f62a806ec80 2024-03-29T11:00:09.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:09.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:09.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:09.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62a77fe640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:00:09.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.849+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6298003520 con 0x7f62a806ec80 2024-03-29T11:00:09.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.850+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 301800768 0 0) 0x7f6298003520 con 0x7f62a806ec80 2024-03-29T11:00:09.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.850+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 >> v1:172.21.15.45:6790/0 conn(0x7f62a806fee0 legacy=0x7f62a813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.850+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 >> v1:172.21.15.45:6789/0 conn(0x7f62a813f3f0 legacy=0x7f62a814a520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:09.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:09.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 1 monclient: found mon.noname-c 2024-03-29T11:00:09.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:09.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a80722f0 con 0x7f62a806ec80 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f629c001690 con 0x7f62a806ec80 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62a77fe640 10 monclient: dump: 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:09.832 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:09.833 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:09.833 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:09.833 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:09.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62ada2d640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:00:09.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.851+0000 7f62ada2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f629c001860 con 0x7f62a806ec80 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62a77fe640 1 -- 172.21.15.45:0/626765130 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f629c001cb0 con 0x7f62a806ec80 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62a77fe640 10 monclient: dump: 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.834 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62ada2d640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.853+0000 7f62ada2d640 4 set_mon_vals no callback set 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 10 monclient: shutdown 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 1 -- 172.21.15.45:0/626765130 >> v1:172.21.15.116:6789/0 conn(0x7f62a806ec80 legacy=0x7f62a80ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 1 -- 172.21.15.45:0/626765130 shutdown_connections 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 1 -- 172.21.15.45:0/626765130 >> 172.21.15.45:0/626765130 conn(0x7f62a8067420 msgr2=0x7f62a80b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 1 -- 172.21.15.45:0/626765130 shutdown_connections 2024-03-29T11:00:09.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 1 -- 172.21.15.45:0/626765130 wait complete. 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 10 monclient: build_initial_monmap 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.854+0000 7f62ada2d640 10 monclient: monmap: 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:09.840 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 1 librados: starting msgr at 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 1 librados: starting objecter 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 1 Processor -- start 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 1 -- start start 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 1 librados: setting wanted keys 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 1 librados: calling monclient init 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.855+0000 7f62ada2d640 10 monclient: init 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62ada2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62ada2d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:09.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62ada2d640 10 monclient(hunting): picked mon.noname-c con 0x7f62a806ec80 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62aca2b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f62a806ec80 0x7f62a81da620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35622/0 (socket says 172.21.15.45:35622) 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62aca2b640 1 -- 172.21.15.45:0/2659994717 learned_addr learned my addr 172.21.15.45:0/2659994717 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62a57fa640 10 client.?.objecter ms_handle_connect 0x7f62a806ec80 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.856+0000 7f62ada2d640 10 monclient(hunting): picked mon.noname-b con 0x7f62a806fee0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62ada2d640 10 monclient(hunting): picked mon.noname-a con 0x7f62a813f3f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a81db180 con 0x7f62a813f3f0 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a8140130 con 0x7f62a806fee0 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a81dad30 con 0x7f62a806ec80 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62ada2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62ada2d640 10 monclient(hunting): authenticate will time out at 1284.126594s 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 10 client.?.objecter ms_handle_connect 0x7f62a806fee0 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:09.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436032792 0 0) 0x7f62a8140130 con 0x7f62a806fee0 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.857+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c004eb0 con 0x7f62a806fee0 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 943151333 0 0) 0x7f628c004eb0 con 0x7f62a806fee0 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f628c005300 con 0x7f62a806fee0 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6294001930 con 0x7f62a806fee0 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): dump: 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:09.843 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955601512 0 0) 0x7f62a81dad30 con 0x7f62a806ec80 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c005850 con 0x7f62a806ec80 2024-03-29T11:00:09.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3871676640 0 0) 0x7f628c005300 con 0x7f62a806fee0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 >> v1:172.21.15.45:6790/0 conn(0x7f62a806ec80 legacy=0x7f62a81da620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.858+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 >> v1:172.21.15.45:6789/0 conn(0x7f62a813f3f0 legacy=0x7f62a81412b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 1 monclient: found mon.b 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a81daec0 con 0x7f62a806fee0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: _finish_auth 0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 10 monclient: _renew_subs 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a814a7a0 con 0x7f62a806fee0 2024-03-29T11:00:09.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:09.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 10 monclient: _renew_subs 2024-03-29T11:00:09.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:09.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a81e2b00 con 0x7f62a806fee0 2024-03-29T11:00:09.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62ada2d640 1 librados: init done 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62940035f0 con 0x7f62a806fee0 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f6294001930 con 0x7f62a806fee0 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.860+0000 7f62a57fa640 10 monclient: dump: 2024-03-29T11:00:09.847 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.861+0000 7f62a6ffd640 4 set_mon_vals no callback set 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.862+0000 7f62ada2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.862+0000 7f62ada2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.862+0000 7f62ada2d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.862+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6288005080 con 0x7f62a806fee0 2024-03-29T11:00:09.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.862+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f62940195d0 con 0x7f62a806fee0 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.863+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f62940835c0 con 0x7f62a806fee0 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.863+0000 7f62a57fa640 10 client.4513.objecter ms_dispatch 0x7f62a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.863+0000 7f62a57fa640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.863+0000 7f62a57fa640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.863+0000 7f62a57fa640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.863+0000 7f62aca2b640 10 monclient: get_auth_request con 0x7f628c06e2e0 auth_method 0 2024-03-29T11:00:09.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.864+0000 7f62a57fa640 10 client.4513.objecter ms_handle_connect 0x7f628c06e2e0 2024-03-29T11:00:09.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.871+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f629404eea0 con 0x7f62a806fee0 2024-03-29T11:00:09.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.871+0000 7f62a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:09.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:09.871+0000 7f62a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:10.154 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.174+0000 7f62ada2d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.174+0000 7f62ada2d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.174+0000 7f62ada2d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:10.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.174+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6288005330 con 0x7f62a806fee0 2024-03-29T11:00:10.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.177+0000 7f62a57fa640 1 -- 172.21.15.45:0/2659994717 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f6294050dd0 con 0x7f62a806fee0 2024-03-29T11:00:10.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.177+0000 7f62a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.177+0000 7f62a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:10.157 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:10.157 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:10.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 10 librados: watch_flush enter 2024-03-29T11:00:10.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 10 librados: watch_flush exit 2024-03-29T11:00:10.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:00:10.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 >> v1:172.21.15.116:6816/3290579615 conn(0x7f628c06e2e0 legacy=0x7f628c0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 10 monclient: shutdown 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 >> v1:172.21.15.116:6789/0 conn(0x7f62a806fee0 legacy=0x7f62a8140ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 shutdown_connections 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 >> 172.21.15.45:0/2659994717 conn(0x7f62a8067420 msgr2=0x7f62a80b1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 shutdown_connections 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 -- 172.21.15.45:0/2659994717 wait complete. 2024-03-29T11:00:10.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:10.180+0000 7f62ada2d640 1 librados: shutdown 2024-03-29T11:00:10.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:16.981 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:17.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.149+0000 7f96cb27b640 10 monclient: get_monmap_and_config 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.149+0000 7f96cb27b640 10 monclient: build_initial_monmap 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.149+0000 7f96cb27b640 10 monclient: monmap: 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:17.150663+0000 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:17.150663+0000 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:17.130 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:17.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.151+0000 7f96cb27b640 1 Processor -- start 2024-03-29T11:00:17.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.151+0000 7f96cb27b640 1 -- start start 2024-03-29T11:00:17.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.151+0000 7f96cb27b640 10 monclient: init 2024-03-29T11:00:17.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.152+0000 7f96cb27b640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:17.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.152+0000 7f96cb27b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:17.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 10 monclient(hunting): picked mon.noname-a con 0x7f96c41457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 10 monclient(hunting): picked mon.noname-b con 0x7f96c413f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 10 monclient(hunting): picked mon.noname-c con 0x7f96c40a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c40aaef0 con 0x7f96c41457f0 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c4145bd0 con 0x7f96c40a9d70 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c4145eb0 con 0x7f96c413f650 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:17.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.154+0000 7f96cb27b640 10 monclient(hunting): authenticate will time out at 1291.423660s 2024-03-29T11:00:17.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96c9017640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f96c41457f0 0x7f96c413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36981/0 (socket says 172.21.15.45:36981) 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96c9017640 1 -- 172.21.15.45:0/3338688984 learned_addr learned my addr 172.21.15.45:0/3338688984 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862020852 0 0) 0x7f96c40aaef0 con 0x7f96c41457f0 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96bbfff640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.155+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b0003520 con 0x7f96c41457f0 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.156+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156559121 0 0) 0x7f96b0003520 con 0x7f96c41457f0 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.156+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 >> v1:172.21.15.45:6790/0 conn(0x7f96c413f650 legacy=0x7f96c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:17.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.156+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 >> v1:172.21.15.116:6789/0 conn(0x7f96c40a9d70 legacy=0x7f96c414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:17.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:17.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 1 monclient: found mon.noname-a 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4146340 con 0x7f96c41457f0 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f96b4001690 con 0x7f96c41457f0 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.159+0000 7f96bbfff640 10 monclient: dump: 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.141 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b4001920 con 0x7f96c41457f0 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96cb27b640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96cb27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96bbfff640 1 -- 172.21.15.45:0/3338688984 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f96b4001cf0 con 0x7f96c41457f0 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96cb27b640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:17.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96cb27b640 4 set_mon_vals no callback set 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.160+0000 7f96bbfff640 10 monclient: dump: 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:17.143 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:17.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.161+0000 7f96cb27b640 10 monclient: shutdown 2024-03-29T11:00:17.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.161+0000 7f96cb27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:17.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.161+0000 7f96cb27b640 1 -- 172.21.15.45:0/3338688984 >> v1:172.21.15.45:6789/0 conn(0x7f96c41457f0 legacy=0x7f96c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.161+0000 7f96cb27b640 1 -- 172.21.15.45:0/3338688984 shutdown_connections 2024-03-29T11:00:17.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.161+0000 7f96cb27b640 1 -- 172.21.15.45:0/3338688984 >> 172.21.15.45:0/3338688984 conn(0x7f96c4066a00 msgr2=0x7f96c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:17.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.163+0000 7f96cb27b640 1 -- 172.21.15.45:0/3338688984 shutdown_connections 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.163+0000 7f96cb27b640 1 -- 172.21.15.45:0/3338688984 wait complete. 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.164+0000 7f96cb27b640 10 monclient: build_initial_monmap 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.164+0000 7f96cb27b640 10 monclient: monmap: 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:17.145 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 1 librados: starting msgr at 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 1 librados: starting objecter 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 1 Processor -- start 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 1 -- start start 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 1 librados: setting wanted keys 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 1 librados: calling monclient init 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.165+0000 7f96cb27b640 10 monclient: init 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient(hunting): picked mon.noname-b con 0x7f96c40a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient(hunting): picked mon.noname-a con 0x7f96c413f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient(hunting): picked mon.noname-c con 0x7f96c4144a40 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c406e830 con 0x7f96c413f650 2024-03-29T11:00:17.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c41df9b0 con 0x7f96c40a9d70 2024-03-29T11:00:17.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c406eaa0 con 0x7f96c4144a40 2024-03-29T11:00:17.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:17.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96cb27b640 10 monclient(hunting): authenticate will time out at 1291.435547s 2024-03-29T11:00:17.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96c9818640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f96c4144a40 0x7f96c41df2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60772/0 (socket says 172.21.15.45:60772) 2024-03-29T11:00:17.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96c9818640 1 -- 172.21.15.45:0/424154153 learned_addr learned my addr 172.21.15.45:0/424154153 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96b9ffb640 10 client.?.objecter ms_handle_connect 0x7f96c4144a40 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.166+0000 7f96b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646465135 0 0) 0x7f96c406eaa0 con 0x7f96c4144a40 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a8004eb0 con 0x7f96c4144a40 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567440907 0 0) 0x7f96a8004eb0 con 0x7f96c4144a40 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96c406eaa0 con 0x7f96c4144a40 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f96c00017f0 con 0x7f96c4144a40 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:17.149 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1249881187 0 0) 0x7f96c406eaa0 con 0x7f96c4144a40 2024-03-29T11:00:17.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.167+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 >> v1:172.21.15.116:6789/0 conn(0x7f96c40a9d70 legacy=0x7f96c4142590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 >> v1:172.21.15.45:6789/0 conn(0x7f96c413f650 legacy=0x7f96c41deb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 1 monclient: found mon.c 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c406ed80 con 0x7f96c4144a40 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:17.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 10 monclient: _renew_subs 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c406f060 con 0x7f96c4144a40 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 10 monclient: _renew_subs 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c41378f0 con 0x7f96c4144a40 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.170+0000 7f96cb27b640 1 librados: init done 2024-03-29T11:00:17.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96c0001dc0 con 0x7f96c4144a40 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f96c0004aa0 con 0x7f96c4144a40 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96b9ffb640 10 monclient: dump: 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:17.153 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:17.154 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.171+0000 7f96bb7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.172+0000 7f96cb27b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.173+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f96c001aa70 con 0x7f96c4144a40 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.174+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f96c0084ae0 con 0x7f96c4144a40 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.174+0000 7f96b9ffb640 10 client.4544.objecter ms_dispatch 0x7f96c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.174+0000 7f96b9ffb640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.174+0000 7f96b9ffb640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.174+0000 7f96b9ffb640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.175+0000 7f96cb27b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.175+0000 7f96cb27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:17.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.175+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a4005080 con 0x7f96c4144a40 2024-03-29T11:00:17.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.176+0000 7f96c9017640 10 monclient: get_auth_request con 0x7f96a806dcd0 auth_method 0 2024-03-29T11:00:17.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.177+0000 7f96b9ffb640 10 client.4544.objecter ms_handle_connect 0x7f96a806dcd0 2024-03-29T11:00:17.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.186+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96c00501f0 con 0x7f96c4144a40 2024-03-29T11:00:17.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.186+0000 7f96b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.186+0000 7f96b9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:17.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.521+0000 7f96cb27b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.521+0000 7f96cb27b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.521+0000 7f96cb27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:17.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.521+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96a4005330 con 0x7f96c4144a40 2024-03-29T11:00:17.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.523+0000 7f96b9ffb640 1 -- 172.21.15.45:0/424154153 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f96c00522f0 con 0x7f96c4144a40 2024-03-29T11:00:17.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.523+0000 7f96b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.523+0000 7f96b9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:17.503 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:17.503 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.526+0000 7f96cb27b640 10 librados: watch_flush enter 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.526+0000 7f96cb27b640 10 librados: watch_flush exit 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.526+0000 7f96cb27b640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.526+0000 7f96cb27b640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.527+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 >> v1:172.21.15.116:6816/3290579615 conn(0x7f96a806dcd0 legacy=0x7f96a8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.527+0000 7f96cb27b640 10 monclient: shutdown 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.527+0000 7f96cb27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:17.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.527+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 >> v1:172.21.15.45:6790/0 conn(0x7f96c4144a40 legacy=0x7f96c41df2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.527+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 shutdown_connections 2024-03-29T11:00:17.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.527+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 >> 172.21.15.45:0/424154153 conn(0x7f96c4066a00 msgr2=0x7f96c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:17.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.528+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 shutdown_connections 2024-03-29T11:00:17.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.529+0000 7f96cb27b640 1 -- 172.21.15.45:0/424154153 wait complete. 2024-03-29T11:00:17.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:17.529+0000 7f96cb27b640 1 librados: shutdown 2024-03-29T11:00:17.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:24.334 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.502+0000 7f5e6e72b640 10 monclient: get_monmap_and_config 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.502+0000 7f5e6e72b640 10 monclient: build_initial_monmap 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.502+0000 7f5e6e72b640 10 monclient: monmap: 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:24.502922+0000 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:24.502922+0000 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:24.482 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:24.483 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:24.483 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:24.483 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:24.483 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:24.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.503+0000 7f5e6e72b640 1 Processor -- start 2024-03-29T11:00:24.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.503+0000 7f5e6e72b640 1 -- start start 2024-03-29T11:00:24.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.503+0000 7f5e6e72b640 10 monclient: init 2024-03-29T11:00:24.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.504+0000 7f5e6e72b640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:24.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.504+0000 7f5e6e72b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:24.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.504+0000 7f5e6e72b640 10 monclient(hunting): picked mon.noname-b con 0x7f5e60038c60 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.507+0000 7f5e6e72b640 10 monclient(hunting): picked mon.noname-a con 0x7f5e600324b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.507+0000 7f5e6e72b640 10 monclient(hunting): picked mon.noname-c con 0x7f5e600d11e0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.507+0000 7f5e6cf28640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5e600324b0 0x7f5e600cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46759/0 (socket says 172.21.15.45:46759) 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.507+0000 7f5e6cf28640 1 -- 172.21.15.45:0/318631671 learned_addr learned my addr 172.21.15.45:0/318631671 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.507+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600caf80 con 0x7f5e600324b0 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.507+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e60039850 con 0x7f5e600d11e0 2024-03-29T11:00:24.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600c78d0 con 0x7f5e60038c60 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e6e72b640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e6e72b640 10 monclient(hunting): authenticate will time out at 1298.777034s 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552992333 0 0) 0x7f5e600caf80 con 0x7f5e600324b0 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e54003520 con 0x7f5e600324b0 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634349699 0 0) 0x7f5e60039850 con 0x7f5e600d11e0 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:00:24.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e600caf80 con 0x7f5e600d11e0 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 433463038 0 0) 0x7f5e54003520 con 0x7f5e600324b0 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 >> v1:172.21.15.45:6790/0 conn(0x7f5e60038c60 legacy=0x7f5e600c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 >> v1:172.21.15.116:6789/0 conn(0x7f5e600d11e0 legacy=0x7f5e600d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 1 monclient: found mon.noname-a 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.508+0000 7f5e5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600c7a60 con 0x7f5e600324b0 2024-03-29T11:00:24.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: _finish_auth 0 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5e58001690 con 0x7f5e600324b0 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: dump: 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.492 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e58001860 con 0x7f5e600324b0 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e6e72b640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e6e72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 1 -- 172.21.15.45:0/318631671 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5e58001c70 con 0x7f5e600324b0 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e5ffff640 10 monclient: dump: 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:24.493 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e6e72b640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.509+0000 7f5e6e72b640 4 set_mon_vals no callback set 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.510+0000 7f5e6e72b640 10 monclient: shutdown 2024-03-29T11:00:24.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.510+0000 7f5e6e72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:24.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.510+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 >> v1:172.21.15.45:6789/0 conn(0x7f5e600324b0 legacy=0x7f5e600cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.511+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 shutdown_connections 2024-03-29T11:00:24.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.511+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 >> 172.21.15.45:0/318631671 conn(0x7f5e60005c20 msgr2=0x7f5e60006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.511+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 shutdown_connections 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.511+0000 7f5e6e72b640 1 -- 172.21.15.45:0/318631671 wait complete. 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.512+0000 7f5e6e72b640 10 monclient: build_initial_monmap 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.513+0000 7f5e6e72b640 10 monclient: monmap: 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:24.498 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.513+0000 7f5e6e72b640 1 librados: starting msgr at 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.513+0000 7f5e6e72b640 1 librados: starting objecter 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.513+0000 7f5e6e72b640 1 Processor -- start 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.514+0000 7f5e6e72b640 1 -- start start 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.514+0000 7f5e6e72b640 1 librados: setting wanted keys 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.514+0000 7f5e6e72b640 1 librados: calling monclient init 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.514+0000 7f5e6e72b640 10 monclient: init 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient(hunting): picked mon.noname-b con 0x7f5e600324b0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient(hunting): picked mon.noname-c con 0x7f5e60038c60 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient(hunting): picked mon.noname-a con 0x7f5e600d11e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:24.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6015c970 con 0x7f5e600d11e0 2024-03-29T11:00:24.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600bea80 con 0x7f5e600324b0 2024-03-29T11:00:24.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600c8c50 con 0x7f5e60038c60 2024-03-29T11:00:24.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:24.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.515+0000 7f5e6e72b640 10 monclient(hunting): authenticate will time out at 1298.784591s 2024-03-29T11:00:24.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.516+0000 7f5e6d729640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5e600324b0 0x7f5e6015ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:55872/0 (socket says 172.21.15.45:55872) 2024-03-29T11:00:24.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.517+0000 7f5e6d729640 1 -- 172.21.15.45:0/3547539631 learned_addr learned my addr 172.21.15.45:0/3547539631 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:24.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.518+0000 7f5e5dffb640 10 client.?.objecter ms_handle_connect 0x7f5e600324b0 2024-03-29T11:00:24.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.518+0000 7f5e5dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 10 client.?.objecter ms_handle_connect 0x7f5e60038c60 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078251164 0 0) 0x7f5e600c8c50 con 0x7f5e60038c60 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.519+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4c004eb0 con 0x7f5e60038c60 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 199793372 0 0) 0x7f5e4c004eb0 con 0x7f5e60038c60 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e600c8c50 con 0x7f5e60038c60 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5e58001b70 con 0x7f5e60038c60 2024-03-29T11:00:24.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 10 monclient(hunting): dump: 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.520+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3729496577 0 0) 0x7f5e600c8c50 con 0x7f5e60038c60 2024-03-29T11:00:24.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 >> v1:172.21.15.116:6789/0 conn(0x7f5e600324b0 legacy=0x7f5e6015ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 >> v1:172.21.15.45:6789/0 conn(0x7f5e600d11e0 legacy=0x7f5e601681b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 1 monclient: found mon.c 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600bc770 con 0x7f5e60038c60 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e6e72b640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e6e72b640 10 monclient: _renew_subs 2024-03-29T11:00:24.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e6e72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e600bee30 con 0x7f5e60038c60 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e6e72b640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.521+0000 7f5e6e72b640 10 monclient: _renew_subs 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e6e72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e600bc520 con 0x7f5e60038c60 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e6e72b640 1 librados: init done 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e58001d70 con 0x7f5e60038c60 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5e58004d90 con 0x7f5e60038c60 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:24.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5dffb640 10 monclient: dump: 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.522+0000 7f5e5f7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:24.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.524+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f5e5801ad60 con 0x7f5e60038c60 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.525+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f5e58084b40 con 0x7f5e60038c60 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.525+0000 7f5e5dffb640 10 client.4550.objecter ms_dispatch 0x7f5e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.525+0000 7f5e5dffb640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.525+0000 7f5e5dffb640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.525+0000 7f5e6d729640 10 monclient: get_auth_request con 0x7f5e4c06dc20 auth_method 0 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.526+0000 7f5e5dffb640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.526+0000 7f5e5dffb640 10 client.4550.objecter ms_handle_connect 0x7f5e4c06dc20 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.527+0000 7f5e6e72b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.527+0000 7f5e6e72b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.527+0000 7f5e6e72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:24.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.527+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48005080 con 0x7f5e60038c60 2024-03-29T11:00:24.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.535+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e58050420 con 0x7f5e60038c60 2024-03-29T11:00:24.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.535+0000 7f5e5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.535+0000 7f5e5dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:24.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.866+0000 7f5e6e72b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:24.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.866+0000 7f5e6e72b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:24.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.866+0000 7f5e6e72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:24.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.866+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e48005330 con 0x7f5e60038c60 2024-03-29T11:00:24.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.870+0000 7f5e5dffb640 1 -- 172.21.15.45:0/3547539631 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f5e58052350 con 0x7f5e60038c60 2024-03-29T11:00:24.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.870+0000 7f5e5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:24.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.870+0000 7f5e5dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:24.850 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:24.851 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:24.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 10 librados: watch_flush enter 2024-03-29T11:00:24.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 10 librados: watch_flush exit 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 >> v1:172.21.15.116:6816/3290579615 conn(0x7f5e4c06dc20 legacy=0x7f5e4c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 10 monclient: shutdown 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.874+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 >> v1:172.21.15.45:6790/0 conn(0x7f5e60038c60 legacy=0x7f5e600bb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.875+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 shutdown_connections 2024-03-29T11:00:24.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.875+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 >> 172.21.15.45:0/3547539631 conn(0x7f5e60005c20 msgr2=0x7f5e600c1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:24.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.876+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 shutdown_connections 2024-03-29T11:00:24.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.876+0000 7f5e6e72b640 1 -- 172.21.15.45:0/3547539631 wait complete. 2024-03-29T11:00:24.856 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:24.876+0000 7f5e6e72b640 1 librados: shutdown 2024-03-29T11:00:24.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:31.679 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:31.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.846+0000 7f30369e8640 10 monclient: get_monmap_and_config 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.846+0000 7f30369e8640 10 monclient: build_initial_monmap 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.846+0000 7f30369e8640 10 monclient: monmap: 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:31.847443+0000 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:31.847443+0000 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:31.827 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:31.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.848+0000 7f30369e8640 1 Processor -- start 2024-03-29T11:00:31.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.848+0000 7f30369e8640 1 -- start start 2024-03-29T11:00:31.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.848+0000 7f30369e8640 10 monclient: init 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient(hunting): picked mon.noname-c con 0x7f303006d9d0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient(hunting): picked mon.noname-b con 0x7f30300b0860 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient(hunting): picked mon.noname-a con 0x7f303013ef30 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300aed10 con 0x7f303013ef30 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300ab1b0 con 0x7f303006d9d0 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300ab490 con 0x7f30300b0860 2024-03-29T11:00:31.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:31.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f30369e8640 10 monclient(hunting): authenticate will time out at 1306.118403s 2024-03-29T11:00:31.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f302f7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f30300b0860 0x7f303013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:45108/0 (socket says 172.21.15.45:45108) 2024-03-29T11:00:31.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.849+0000 7f302f7fe640 1 -- 172.21.15.45:0/3561410457 learned_addr learned my addr 172.21.15.45:0/3561410457 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:31.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.850+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496585321 0 0) 0x7f30300ab490 con 0x7f30300b0860 2024-03-29T11:00:31.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.850+0000 7f302effd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.850+0000 7f302effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.850+0000 7f302effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.850+0000 7f302effd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:00:31.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.850+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3018003520 con 0x7f30300b0860 2024-03-29T11:00:31.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.851+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4181797206 0 0) 0x7f3018003520 con 0x7f30300b0860 2024-03-29T11:00:31.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.851+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 >> v1:172.21.15.116:6789/0 conn(0x7f303006d9d0 legacy=0x7f30300af1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:00:31.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.851+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 >> v1:172.21.15.45:6789/0 conn(0x7f303013ef30 legacy=0x7f303014ab80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:31.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:31.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 1 monclient: found mon.noname-b 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300ab770 con 0x7f30300b0860 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: _finish_auth 0 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3020001690 con 0x7f30300b0860 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f302effd640 10 monclient: dump: 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:31.836 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f30369e8640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.853+0000 7f30369e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30200018f0 con 0x7f30300b0860 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f302effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:31.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f302effd640 1 -- 172.21.15.45:0/3561410457 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3020001d40 con 0x7f30300b0860 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f302effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f302effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f302effd640 10 monclient: dump: 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:31.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f30369e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.854+0000 7f30369e8640 4 set_mon_vals no callback set 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.855+0000 7f30369e8640 10 monclient: shutdown 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.855+0000 7f30369e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.855+0000 7f30369e8640 1 -- 172.21.15.45:0/3561410457 >> v1:172.21.15.45:6790/0 conn(0x7f30300b0860 legacy=0x7f303013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.855+0000 7f30369e8640 1 -- 172.21.15.45:0/3561410457 shutdown_connections 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.855+0000 7f30369e8640 1 -- 172.21.15.45:0/3561410457 >> 172.21.15.45:0/3561410457 conn(0x7f303006b660 msgr2=0x7f303006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.855+0000 7f30369e8640 1 -- 172.21.15.45:0/3561410457 shutdown_connections 2024-03-29T11:00:31.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.856+0000 7f30369e8640 1 -- 172.21.15.45:0/3561410457 wait complete. 2024-03-29T11:00:31.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.856+0000 7f30369e8640 10 monclient: build_initial_monmap 2024-03-29T11:00:31.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.856+0000 7f30369e8640 10 monclient: monmap: 2024-03-29T11:00:31.841 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.857+0000 7f30369e8640 1 librados: starting msgr at 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.857+0000 7f30369e8640 1 librados: starting objecter 2024-03-29T11:00:31.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.857+0000 7f30369e8640 1 Processor -- start 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.858+0000 7f30369e8640 1 -- start start 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.858+0000 7f30369e8640 1 librados: setting wanted keys 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.858+0000 7f30369e8640 1 librados: calling monclient init 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.858+0000 7f30369e8640 10 monclient: init 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.859+0000 7f30369e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.859+0000 7f30369e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.859+0000 7f30369e8640 10 monclient(hunting): picked mon.noname-a con 0x7f303006d9d0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 10 monclient(hunting): picked mon.noname-b con 0x7f30300b0860 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 10 monclient(hunting): picked mon.noname-c con 0x7f303013ef30 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303006e6c0 con 0x7f303006d9d0 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303006e930 con 0x7f30300b0860 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303006ec10 con 0x7f303013ef30 2024-03-29T11:00:31.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:31.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f302f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f30300b0860 0x7f303006f030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:48086/0 (socket says 172.21.15.45:48086) 2024-03-29T11:00:31.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f302f7fe640 1 -- 172.21.15.45:0/1723377813 learned_addr learned my addr 172.21.15.45:0/1723377813 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:31.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.860+0000 7f30369e8640 10 monclient(hunting): authenticate will time out at 1306.129834s 2024-03-29T11:00:31.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 10 client.?.objecter ms_handle_connect 0x7f30300b0860 2024-03-29T11:00:31.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:31.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902389737 0 0) 0x7f303006e930 con 0x7f30300b0860 2024-03-29T11:00:31.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:00:31.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.861+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c004eb0 con 0x7f30300b0860 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451676041 0 0) 0x7f301c004eb0 con 0x7f30300b0860 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f303006e930 con 0x7f30300b0860 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3020001650 con 0x7f30300b0860 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): dump: 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:31.846 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:31.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 client.?.objecter ms_handle_connect 0x7f303013ef30 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926093663 0 0) 0x7f303006ec10 con 0x7f303013ef30 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.862+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c004eb0 con 0x7f303013ef30 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3845304677 0 0) 0x7f303006e930 con 0x7f30300b0860 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 >> v1:172.21.15.45:6790/0 conn(0x7f303013ef30 legacy=0x7f30300b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 >> v1:172.21.15.45:6789/0 conn(0x7f303006d9d0 legacy=0x7f3030137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 monclient: found mon.b 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300b1e90 con 0x7f30300b0860 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 10 monclient: _renew_subs 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30300b3240 con 0x7f30300b0860 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 10 monclient: _renew_subs 2024-03-29T11:00:31.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3020004ce0 con 0x7f30300b0860 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30300b2f60 con 0x7f30300b0860 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f30369e8640 1 librados: init done 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f30200050b0 con 0x7f30300b0860 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302cff9640 10 monclient: dump: 2024-03-29T11:00:31.850 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:31.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.863+0000 7f302e7fc640 4 set_mon_vals no callback set 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f30369e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f30369e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f30369e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3030075f10 con 0x7f30300b0860 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f3020005210 con 0x7f30300b0860 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f3020084d80 con 0x7f30300b0860 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f302cff9640 10 client.4522.objecter ms_dispatch 0x7f3030000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:31.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f302cff9640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f302cff9640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.867+0000 7f302cff9640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.868+0000 7f302ffff640 10 monclient: get_auth_request con 0x7f301c06dd60 auth_method 0 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.869+0000 7f302cff9640 10 client.4522.objecter ms_handle_connect 0x7f301c06dd60 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.874+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3020050660 con 0x7f30300b0860 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.874+0000 7f302cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:31.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:31.874+0000 7f302cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:32.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.188+0000 7f30369e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.188+0000 7f30369e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.188+0000 7f30369e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:32.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.188+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3030072330 con 0x7f30300b0860 2024-03-29T11:00:32.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.189+0000 7f302cff9640 1 -- 172.21.15.45:0/1723377813 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f3020052590 con 0x7f30300b0860 2024-03-29T11:00:32.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.189+0000 7f302cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.189+0000 7f302cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:32.169 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:32.169 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:32.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 10 librados: watch_flush enter 2024-03-29T11:00:32.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 10 librados: watch_flush exit 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 >> v1:172.21.15.116:6816/3290579615 conn(0x7f301c06dd60 legacy=0x7f301c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 10 monclient: shutdown 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 >> v1:172.21.15.116:6789/0 conn(0x7f30300b0860 legacy=0x7f303006f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 shutdown_connections 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 >> 172.21.15.45:0/1723377813 conn(0x7f303006b660 msgr2=0x7f30301cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 shutdown_connections 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.192+0000 7f30369e8640 1 -- 172.21.15.45:0/1723377813 wait complete. 2024-03-29T11:00:32.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:32.193+0000 7f30369e8640 1 librados: shutdown 2024-03-29T11:00:32.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:38.994 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:39.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.175+0000 7fb6d136c640 10 monclient: get_monmap_and_config 2024-03-29T11:00:39.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.175+0000 7fb6d136c640 10 monclient: build_initial_monmap 2024-03-29T11:00:39.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.175+0000 7fb6d136c640 10 monclient: monmap: 2024-03-29T11:00:39.155 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:39.176457+0000 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:39.176457+0000 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:39.156 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.176+0000 7fb6d136c640 1 Processor -- start 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.176+0000 7fb6d136c640 1 -- start start 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.176+0000 7fb6d136c640 10 monclient: init 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.177+0000 7fb6d136c640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.177+0000 7fb6d136c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.177+0000 7fb6d136c640 10 monclient(hunting): picked mon.noname-a con 0x7fb6cc1422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:39.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.177+0000 7fb6d136c640 10 monclient(hunting): picked mon.noname-c con 0x7fb6cc142690 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:39.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6d136c640 10 monclient(hunting): picked mon.noname-b con 0x7fb6cc0b17d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6d136c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc142c80 con 0x7fb6cc1422b0 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6d136c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc0aae40 con 0x7fb6cc142690 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6ca7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb6cc142690 0x7fb6cc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:37738/0 (socket says 172.21.15.45:37738) 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6ca7fc640 1 -- 172.21.15.45:0/241215668 learned_addr learned my addr 172.21.15.45:0/241215668 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6d136c640 1 -- 172.21.15.45:0/241215668 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc13fbd0 con 0x7fb6cc0b17d0 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6d136c640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6d136c640 10 monclient(hunting): authenticate will time out at 1313.447377s 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286924139 0 0) 0x7fb6cc0aae40 con 0x7fb6cc142690 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6c9ffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:00:39.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.178+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b4003520 con 0x7fb6cc142690 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081229805 0 0) 0x7fb6cc13fbd0 con 0x7fb6cc0b17d0 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6cc0aae40 con 0x7fb6cc0b17d0 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751553764 0 0) 0x7fb6b4003520 con 0x7fb6cc142690 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 >> v1:172.21.15.45:6790/0 conn(0x7fb6cc0b17d0 legacy=0x7fb6cc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 >> v1:172.21.15.45:6789/0 conn(0x7fb6cc1422b0 legacy=0x7fb6cc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc14f920 con 0x7fb6cc142690 2024-03-29T11:00:39.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb6c0001690 con 0x7fb6cc142690 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: dump: 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:39.163 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6d136c640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6d136c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 1 -- 172.21.15.45:0/241215668 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6c00017f0 con 0x7fb6cc142690 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6d136c640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.179+0000 7fb6d136c640 4 set_mon_vals no callback set 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.180+0000 7fb6d136c640 10 monclient: shutdown 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.180+0000 7fb6d136c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.180+0000 7fb6d136c640 1 -- 172.21.15.45:0/241215668 >> v1:172.21.15.116:6789/0 conn(0x7fb6cc142690 legacy=0x7fb6cc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.180+0000 7fb6d136c640 1 -- 172.21.15.45:0/241215668 shutdown_connections 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.180+0000 7fb6d136c640 1 -- 172.21.15.45:0/241215668 >> 172.21.15.45:0/241215668 conn(0x7fb6cc066a00 msgr2=0x7fb6cc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:39.164 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.181+0000 7fb6d136c640 1 -- 172.21.15.45:0/241215668 shutdown_connections 2024-03-29T11:00:39.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.181+0000 7fb6d136c640 1 -- 172.21.15.45:0/241215668 wait complete. 2024-03-29T11:00:39.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.182+0000 7fb6d136c640 10 monclient: build_initial_monmap 2024-03-29T11:00:39.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.182+0000 7fb6d136c640 10 monclient: monmap: 2024-03-29T11:00:39.166 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:39.166 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:39.166 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 1 librados: starting msgr at 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 1 librados: starting objecter 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 1 Processor -- start 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 1 -- start start 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 1 librados: setting wanted keys 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 1 librados: calling monclient init 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.183+0000 7fb6d136c640 10 monclient: init 2024-03-29T11:00:39.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient(hunting): picked mon.noname-a con 0x7fb6cc0b17d0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient(hunting): picked mon.noname-c con 0x7fb6cc1422b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient(hunting): picked mon.noname-b con 0x7fb6cc142690 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc12ccf0 con 0x7fb6cc0b17d0 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc0719e0 con 0x7fb6cc142690 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc12cf90 con 0x7fb6cc1422b0 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6d136c640 10 monclient(hunting): authenticate will time out at 1313.454367s 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6cb7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb6cc142690 0x7fb6cc06f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:38376/0 (socket says 172.21.15.45:38376) 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6cb7fe640 1 -- 172.21.15.45:0/3826869226 learned_addr learned my addr 172.21.15.45:0/3826869226 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6b37fe640 10 client.?.objecter ms_handle_connect 0x7fb6cc142690 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.185+0000 7fb6b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 client.?.objecter ms_handle_connect 0x7fb6cc1422b0 2024-03-29T11:00:39.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 client.?.objecter ms_handle_connect 0x7fb6cc0b17d0 2024-03-29T11:00:39.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376972450 0 0) 0x7fb6cc0719e0 con 0x7fb6cc142690 2024-03-29T11:00:39.170 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.186+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b8004eb0 con 0x7fb6cc142690 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564991858 0 0) 0x7fb6cc12cf90 con 0x7fb6cc1422b0 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6cc0719e0 con 0x7fb6cc1422b0 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830332820 0 0) 0x7fb6cc12ccf0 con 0x7fb6cc0b17d0 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.171 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6cc12cf90 con 0x7fb6cc0b17d0 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1805721245 0 0) 0x7fb6b8004eb0 con 0x7fb6cc142690 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6cc12ccf0 con 0x7fb6cc142690 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb6c4001c90 con 0x7fb6cc142690 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): dump: 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:39.172 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 66780444 0 0) 0x7fb6cc12ccf0 con 0x7fb6cc142690 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 >> v1:172.21.15.45:6790/0 conn(0x7fb6cc1422b0 legacy=0x7fb6cc12c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.187+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 >> v1:172.21.15.45:6789/0 conn(0x7fb6cc0b17d0 legacy=0x7fb6cc12f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 1 monclient: found mon.b 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:39.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc070050 con 0x7fb6cc142690 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 10 monclient: _renew_subs 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc070320 con 0x7fb6cc142690 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 10 monclient: _renew_subs 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc070620 con 0x7fb6cc142690 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.188+0000 7fb6d136c640 1 librados: init done 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6c4001df0 con 0x7fb6cc142690 2024-03-29T11:00:39.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fb6c4003bd0 con 0x7fb6cc142690 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6b37fe640 10 monclient: dump: 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:39.175 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.189+0000 7fb6c8ff9640 4 set_mon_vals no callback set 2024-03-29T11:00:39.176 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.190+0000 7fb6b17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.191+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fb6c4003d30 con 0x7fb6cc142690 2024-03-29T11:00:39.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:39.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc075f10 con 0x7fb6cc142690 2024-03-29T11:00:39.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fb6c4084d70 con 0x7fb6cc142690 2024-03-29T11:00:39.177 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6caffd640 10 monclient: get_auth_request con 0x7fb6b806de50 auth_method 0 2024-03-29T11:00:39.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b37fe640 10 client.4528.objecter ms_dispatch 0x7fb6cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:39.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b37fe640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:39.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b37fe640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:39.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.192+0000 7fb6b37fe640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:00:39.178 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.193+0000 7fb6b37fe640 10 client.4528.objecter ms_handle_connect 0x7fb6b806de50 2024-03-29T11:00:39.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.200+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6c4050480 con 0x7fb6cc142690 2024-03-29T11:00:39.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.200+0000 7fb6b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.200+0000 7fb6b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:39.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.642+0000 7fb6b17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.642+0000 7fb6b17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.642+0000 7fb6b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:00:39.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.642+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6cc0b2270 con 0x7fb6cc142690 2024-03-29T11:00:39.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.643+0000 7fb6b37fe640 1 -- 172.21.15.45:0/3826869226 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fb6c4052580 con 0x7fb6cc142690 2024-03-29T11:00:39.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.643+0000 7fb6b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.643+0000 7fb6b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:39.623 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:39.623 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:39.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 10 librados: watch_flush enter 2024-03-29T11:00:39.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 10 librados: watch_flush exit 2024-03-29T11:00:39.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:00:39.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 >> v1:172.21.15.116:6816/3290579615 conn(0x7fb6b806de50 legacy=0x7fb6b8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 10 monclient: shutdown 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 >> v1:172.21.15.116:6789/0 conn(0x7fb6cc142690 legacy=0x7fb6cc06f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.647+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 shutdown_connections 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.648+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 >> 172.21.15.45:0/3826869226 conn(0x7fb6cc066a00 msgr2=0x7fb6cc13c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.648+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 shutdown_connections 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.648+0000 7fb6b17fa640 1 -- 172.21.15.45:0/3826869226 wait complete. 2024-03-29T11:00:39.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:39.648+0000 7fb6b17fa640 1 librados: shutdown 2024-03-29T11:00:39.643 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:46.450 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:46.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.620+0000 7f473c17b640 10 monclient: get_monmap_and_config 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.620+0000 7f473c17b640 10 monclient: build_initial_monmap 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.620+0000 7f473c17b640 10 monclient: monmap: 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:46.622611+0000 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:46.622611+0000 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:46.602 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:46.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.623+0000 7f473c17b640 1 Processor -- start 2024-03-29T11:00:46.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.625+0000 7f473c17b640 1 -- start start 2024-03-29T11:00:46.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.625+0000 7f473c17b640 10 monclient: init 2024-03-29T11:00:46.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:46.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:46.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient(hunting): picked mon.noname-a con 0x7f47340b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:46.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient(hunting): picked mon.noname-b con 0x7f473414a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:46.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient(hunting): picked mon.noname-c con 0x7f47340ac5a0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:46.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340aba70 con 0x7f47340b0770 2024-03-29T11:00:46.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340aecb0 con 0x7f47340ac5a0 2024-03-29T11:00:46.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340af150 con 0x7f473414a700 2024-03-29T11:00:46.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:46.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.626+0000 7f473c17b640 10 monclient(hunting): authenticate will time out at 1320.896362s 2024-03-29T11:00:46.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.629+0000 7f473a718640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f47340ac5a0 0x7f4734151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:46971/0 (socket says 172.21.15.45:46971) 2024-03-29T11:00:46.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.629+0000 7f473a718640 1 -- 172.21.15.45:0/1251861345 learned_addr learned my addr 172.21.15.45:0/1251861345 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583344616 0 0) 0x7f47340af150 con 0x7f473414a700 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4720003520 con 0x7f473414a700 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122650128 0 0) 0x7f47340aecb0 con 0x7f47340ac5a0 2024-03-29T11:00:46.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47340af150 con 0x7f47340ac5a0 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3463522090 0 0) 0x7f47340af150 con 0x7f47340ac5a0 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 >> v1:172.21.15.45:6790/0 conn(0x7f473414a700 legacy=0x7f47340abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.632+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 >> v1:172.21.15.45:6789/0 conn(0x7f47340b0770 legacy=0x7f4734149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: _finish_hunting 0 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 1 monclient: found mon.noname-c 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340ac9d0 con 0x7f47340ac5a0 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: _finish_auth 0 2024-03-29T11:00:46.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: _check_auth_tickets 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f472c0019f0 con 0x7f47340ac5a0 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.635+0000 7f4738f15640 10 monclient: dump: 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:46.617 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.636+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f472c001b50 con 0x7f47340ac5a0 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.636+0000 7f4738f15640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.636+0000 7f4738f15640 1 -- 172.21.15.45:0/1251861345 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f472c001f60 con 0x7f47340ac5a0 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.636+0000 7f4738f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.636+0000 7f4738f15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.636+0000 7f4738f15640 10 monclient: dump: 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:46.618 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:46.619 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.637+0000 7f473c17b640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.637+0000 7f473c17b640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.637+0000 7f473c17b640 4 set_mon_vals no callback set 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.638+0000 7f473c17b640 10 monclient: shutdown 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.638+0000 7f473c17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.638+0000 7f473c17b640 1 -- 172.21.15.45:0/1251861345 >> v1:172.21.15.116:6789/0 conn(0x7f47340ac5a0 legacy=0x7f4734151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.639+0000 7f473c17b640 1 -- 172.21.15.45:0/1251861345 shutdown_connections 2024-03-29T11:00:46.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.639+0000 7f473c17b640 1 -- 172.21.15.45:0/1251861345 >> 172.21.15.45:0/1251861345 conn(0x7f473406b660 msgr2=0x7f473406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:46.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.644+0000 7f473c17b640 1 -- 172.21.15.45:0/1251861345 shutdown_connections 2024-03-29T11:00:46.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.644+0000 7f473c17b640 1 -- 172.21.15.45:0/1251861345 wait complete. 2024-03-29T11:00:46.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.646+0000 7f473c17b640 10 monclient: build_initial_monmap 2024-03-29T11:00:46.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.646+0000 7f473c17b640 10 monclient: monmap: 2024-03-29T11:00:46.627 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:46.627 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:46.627 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:46.627 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.646+0000 7f473c17b640 1 librados: starting msgr at 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.646+0000 7f473c17b640 1 librados: starting objecter 2024-03-29T11:00:46.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.646+0000 7f473c17b640 1 Processor -- start 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 1 -- start start 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 1 librados: setting wanted keys 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 1 librados: calling monclient init 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient: init 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient(hunting): picked mon.noname-a con 0x7f47340ac5a0 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:46.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient(hunting): picked mon.noname-b con 0x7f47340b0770 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient(hunting): picked mon.noname-c con 0x7f473414a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473411f390 con 0x7f47340ac5a0 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473411f600 con 0x7f47340b0770 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47341e7c70 con 0x7f473414a700 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.647+0000 7f473c17b640 10 monclient(hunting): authenticate will time out at 1320.917930s 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.651+0000 7f4739f17640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f47340ac5a0 0x7f47341e7410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46009/0 (socket says 172.21.15.45:46009) 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.651+0000 7f4739f17640 1 -- 172.21.15.45:0/3480156390 learned_addr learned my addr 172.21.15.45:0/3480156390 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 10 client.?.objecter ms_handle_connect 0x7f47340ac5a0 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112884674 0 0) 0x7f473411f390 con 0x7f47340ac5a0 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.652+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4718004eb0 con 0x7f47340ac5a0 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 10 client.?.objecter ms_handle_connect 0x7f473414a700 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614560882 0 0) 0x7f47341e7c70 con 0x7f473414a700 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f473411f390 con 0x7f473414a700 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3288464280 0 0) 0x7f4718004eb0 con 0x7f47340ac5a0 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.653+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47341e7c70 con 0x7f47340ac5a0 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4730001960 con 0x7f47340ac5a0 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient(hunting): dump: 2024-03-29T11:00:46.639 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 668673061 0 0) 0x7f473411f390 con 0x7f473414a700 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4718004eb0 con 0x7f473414a700 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 525904333 0 0) 0x7f47341e7c70 con 0x7f47340ac5a0 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 >> v1:172.21.15.45:6790/0 conn(0x7f473414a700 legacy=0x7f4734147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 >> v1:172.21.15.116:6789/0 conn(0x7f47340b0770 legacy=0x7f473411cf30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:46.640 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 monclient: found mon.a 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47341e8190 con 0x7f47340ac5a0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.654+0000 7f472a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 10 monclient: _renew_subs 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473411d640 con 0x7f47340ac5a0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 10 monclient: _renew_subs 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4734070760 con 0x7f47340ac5a0 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f473c17b640 1 librados: init done 2024-03-29T11:00:46.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4730001ac0 con 0x7f47340ac5a0 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f472a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f47300038a0 con 0x7f47340ac5a0 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f472a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.655+0000 7f472a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.656+0000 7f472a7fc640 10 monclient: dump: 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:46.642 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:46.643 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:46.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.656+0000 7f472b7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f4730003a00 con 0x7f47340ac5a0 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f4730084a00 con 0x7f47340ac5a0 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f472a7fc640 10 client.4512.objecter ms_dispatch 0x7f4734000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f472a7fc640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f472a7fc640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f472a7fc640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.657+0000 7f473c17b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f473c17b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f473c17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c005080 con 0x7f47340ac5a0 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.658+0000 7f4739716640 10 monclient: get_auth_request con 0x7f471806dde0 auth_method 0 2024-03-29T11:00:46.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.659+0000 7f472a7fc640 10 client.4512.objecter ms_handle_connect 0x7f471806dde0 2024-03-29T11:00:46.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.669+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4730050110 con 0x7f47340ac5a0 2024-03-29T11:00:46.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.669+0000 7f472a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:46.669+0000 7f472a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:47.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.031+0000 7f473c17b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.031+0000 7f473c17b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.031+0000 7f473c17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:00:47.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.031+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f471c005330 con 0x7f47340ac5a0 2024-03-29T11:00:47.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.033+0000 7f472a7fc640 1 -- 172.21.15.45:0/3480156390 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f4730052210 con 0x7f47340ac5a0 2024-03-29T11:00:47.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.033+0000 7f472a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.033+0000 7f472a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:47.014 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:47.014 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:47.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 10 librados: watch_flush enter 2024-03-29T11:00:47.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 10 librados: watch_flush exit 2024-03-29T11:00:47.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:00:47.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:00:47.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 >> v1:172.21.15.116:6816/3290579615 conn(0x7f471806dde0 legacy=0x7f47180702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:47.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 10 monclient: shutdown 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 >> v1:172.21.15.45:6789/0 conn(0x7f47340ac5a0 legacy=0x7f47341e7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 shutdown_connections 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 >> 172.21.15.45:0/3480156390 conn(0x7f473406b660 msgr2=0x7f47340b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 shutdown_connections 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.037+0000 7f473c17b640 1 -- 172.21.15.45:0/3480156390 wait complete. 2024-03-29T11:00:47.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:47.038+0000 7f473c17b640 1 librados: shutdown 2024-03-29T11:00:47.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:53.840 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.003+0000 7ff677025640 10 monclient: get_monmap_and_config 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.003+0000 7ff677025640 10 monclient: build_initial_monmap 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.003+0000 7ff677025640 10 monclient: monmap: 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:00:54.004883+0000 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:00:54.004883+0000 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:53.984 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:00:53.985 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:00:53.985 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:53.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.007+0000 7ff677025640 1 Processor -- start 2024-03-29T11:00:53.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.007+0000 7ff677025640 1 -- start start 2024-03-29T11:00:53.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.007+0000 7ff677025640 10 monclient: init 2024-03-29T11:00:53.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:53.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:53.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient(hunting): picked mon.noname-c con 0x7ff6701457f0 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:53.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient(hunting): picked mon.noname-a con 0x7ff67013f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:53.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient(hunting): picked mon.noname-b con 0x7ff6700a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:53.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700aaef0 con 0x7ff67013f650 2024-03-29T11:00:53.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff670145bd0 con 0x7ff6701457f0 2024-03-29T11:00:53.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff670145eb0 con 0x7ff6700a9d70 2024-03-29T11:00:53.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:53.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.008+0000 7ff677025640 10 monclient(hunting): authenticate will time out at 1328.278507s 2024-03-29T11:00:53.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.014+0000 7ff6755c2640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff6700a9d70 0x7ff67014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36757/0 (socket says 172.21.15.45:36757) 2024-03-29T11:00:53.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.014+0000 7ff6755c2640 1 -- 172.21.15.45:0/1804035240 learned_addr learned my addr 172.21.15.45:0/1804035240 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:53.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.015+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593173602 0 0) 0x7ff670145eb0 con 0x7ff6700a9d70 2024-03-29T11:00:53.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.015+0000 7ff6677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:53.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.015+0000 7ff6677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:53.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.015+0000 7ff6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:53.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.015+0000 7ff6677fe640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:00:53.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.015+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff658003520 con 0x7ff6700a9d70 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2524888261 0 0) 0x7ff658003520 con 0x7ff6700a9d70 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 >> v1:172.21.15.116:6789/0 conn(0x7ff6701457f0 legacy=0x7ff67013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 >> v1:172.21.15.45:6789/0 conn(0x7ff67013f650 legacy=0x7ff6700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 1 monclient: found mon.noname-b 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff670146340 con 0x7ff6700a9d70 2024-03-29T11:00:54.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff6677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff677025640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.018+0000 7ff677025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff6680019f0 con 0x7ff6700a9d70 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: dump: 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:54.003 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff677025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff668001c80 con 0x7ff6700a9d70 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff677025640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:54.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff677025640 4 set_mon_vals no callback set 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 1 -- 172.21.15.45:0/1804035240 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff668002010 con 0x7ff6700a9d70 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.019+0000 7ff6677fe640 10 monclient: dump: 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:54.005 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:54.006 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:54.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.024+0000 7ff677025640 10 monclient: shutdown 2024-03-29T11:00:54.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.024+0000 7ff677025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:54.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.024+0000 7ff677025640 1 -- 172.21.15.45:0/1804035240 >> v1:172.21.15.45:6790/0 conn(0x7ff6700a9d70 legacy=0x7ff67014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.026+0000 7ff677025640 1 -- 172.21.15.45:0/1804035240 shutdown_connections 2024-03-29T11:00:54.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.026+0000 7ff677025640 1 -- 172.21.15.45:0/1804035240 >> 172.21.15.45:0/1804035240 conn(0x7ff670066a00 msgr2=0x7ff670066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:54.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.026+0000 7ff677025640 1 -- 172.21.15.45:0/1804035240 shutdown_connections 2024-03-29T11:00:54.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.026+0000 7ff677025640 1 -- 172.21.15.45:0/1804035240 wait complete. 2024-03-29T11:00:54.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.027+0000 7ff677025640 10 monclient: build_initial_monmap 2024-03-29T11:00:54.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.027+0000 7ff677025640 10 monclient: monmap: 2024-03-29T11:00:54.009 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:00:54.009 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 1 librados: starting msgr at 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 1 librados: starting objecter 2024-03-29T11:00:54.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 1 Processor -- start 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 1 -- start start 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 1 librados: setting wanted keys 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 1 librados: calling monclient init 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.028+0000 7ff677025640 10 monclient: init 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient(hunting): picked mon.noname-a con 0x7ff6700a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient(hunting): picked mon.noname-b con 0x7ff67013f650 addr v1:172.21.15.116:6789/0 2024-03-29T11:00:54.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient(hunting): picked mon.noname-c con 0x7ff6701457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:00:54.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67013b080 con 0x7ff6700a9d70 2024-03-29T11:00:54.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67013b2f0 con 0x7ff67013f650 2024-03-29T11:00:54.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67013b5d0 con 0x7ff6701457f0 2024-03-29T11:00:54.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:54.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.029+0000 7ff677025640 10 monclient(hunting): authenticate will time out at 1328.299350s 2024-03-29T11:00:54.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.033+0000 7ff6755c2640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff6701457f0 0x7ff6701e60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36899/0 (socket says 172.21.15.45:36899) 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.033+0000 7ff6755c2640 1 -- 172.21.15.45:0/2561775306 learned_addr learned my addr 172.21.15.45:0/2561775306 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 client.?.objecter ms_handle_connect 0x7ff6701457f0 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533925521 0 0) 0x7ff67013b5d0 con 0x7ff6701457f0 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:54.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c004eb0 con 0x7ff6701457f0 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616638637 0 0) 0x7ff65c004eb0 con 0x7ff6701457f0 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67013b5d0 con 0x7ff6701457f0 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff6680019b0 con 0x7ff6701457f0 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:54.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): dump: 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.034+0000 7ff6657fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:54.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.035+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1057268351 0 0) 0x7ff67013b5d0 con 0x7ff6701457f0 2024-03-29T11:00:54.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.035+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 >> v1:172.21.15.116:6789/0 conn(0x7ff67013f650 legacy=0x7ff6701408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.035+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 >> v1:172.21.15.45:6789/0 conn(0x7ff6700a9d70 legacy=0x7ff670139c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:54.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.038+0000 7ff6657fa640 10 monclient: _finish_hunting 0 2024-03-29T11:00:54.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 1 monclient: found mon.c 2024-03-29T11:00:54.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:54.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67006de90 con 0x7ff6701457f0 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 10 monclient: _finish_auth 0 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 10 monclient: _check_auth_tickets 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff6657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 10 monclient: _renew_subs 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff670135c00 con 0x7ff6701457f0 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 10 monclient: _renew_subs 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:54.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff670139610 con 0x7ff6701457f0 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.040+0000 7ff677025640 1 librados: init done 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff677025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff677025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff677025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff670075f10 con 0x7ff6701457f0 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff668001590 con 0x7ff6701457f0 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff6657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7ff668005110 con 0x7ff6701457f0 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff6657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff6657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff6657fa640 10 monclient: dump: 2024-03-29T11:00:54.024 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:00:54.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.041+0000 7ff666ffd640 4 set_mon_vals no callback set 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.043+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7ff66801b0e0 con 0x7ff6701457f0 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.044+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7ff668084f40 con 0x7ff6701457f0 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.044+0000 7ff6657fa640 10 client.4574.objecter ms_dispatch 0x7ff670000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.044+0000 7ff6657fa640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.044+0000 7ff6657fa640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.044+0000 7ff6657fa640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.045+0000 7ff674dc1640 10 monclient: get_auth_request con 0x7ff65c06dde0 auth_method 0 2024-03-29T11:00:54.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.045+0000 7ff6657fa640 10 client.4574.objecter ms_handle_connect 0x7ff65c06dde0 2024-03-29T11:00:54.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.051+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff668050650 con 0x7ff6701457f0 2024-03-29T11:00:54.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.051+0000 7ff6657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.051+0000 7ff6657fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:54.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.385+0000 7ff677025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.385+0000 7ff677025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.385+0000 7ff677025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:00:54.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.385+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff670135d90 con 0x7ff6701457f0 2024-03-29T11:00:54.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.386+0000 7ff6657fa640 1 -- 172.21.15.45:0/2561775306 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7ff668052750 con 0x7ff6701457f0 2024-03-29T11:00:54.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.386+0000 7ff6657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.386+0000 7ff6657fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:54.367 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:00:54.367 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:54.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.389+0000 7ff677025640 10 librados: watch_flush enter 2024-03-29T11:00:54.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 10 librados: watch_flush exit 2024-03-29T11:00:54.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:00:54.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 >> v1:172.21.15.116:6816/3290579615 conn(0x7ff65c06dde0 legacy=0x7ff65c0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 10 monclient: shutdown 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 >> v1:172.21.15.45:6790/0 conn(0x7ff6701457f0 legacy=0x7ff6701e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 shutdown_connections 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.390+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 >> 172.21.15.45:0/2561775306 conn(0x7ff670066a00 msgr2=0x7ff6700b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.391+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 shutdown_connections 2024-03-29T11:00:54.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.391+0000 7ff677025640 1 -- 172.21.15.45:0/2561775306 wait complete. 2024-03-29T11:00:54.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:00:54.391+0000 7ff677025640 1 librados: shutdown 2024-03-29T11:00:54.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:01.198 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.369+0000 7f53a7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.369+0000 7f53a7fff640 10 monclient: build_initial_monmap 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.369+0000 7f53a7fff640 10 monclient: monmap: 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:01.369938+0000 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:01.369938+0000 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:01.349 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:01.350 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:01.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.370+0000 7f53a7fff640 1 Processor -- start 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.370+0000 7f53a7fff640 1 -- start start 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.370+0000 7f53a7fff640 10 monclient: init 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f53a80b2800 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f53a80a9c00 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f53a80b38c0 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a8141eb0 con 0x7f53a80b2800 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a80b22d0 con 0x7f53a80b38c0 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a8142d30 con 0x7f53a80a9c00 2024-03-29T11:01:01.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:01.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.371+0000 7f53a7fff640 10 monclient(hunting): authenticate will time out at 1335.640337s 2024-03-29T11:01:01.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.378+0000 7f53a67fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f53a80a9c00 0x7f53a813bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43429/0 (socket says 172.21.15.45:43429) 2024-03-29T11:01:01.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.378+0000 7f53a67fc640 1 -- 172.21.15.45:0/1021433381 learned_addr learned my addr 172.21.15.45:0/1021433381 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461035940 0 0) 0x7f53a80b22d0 con 0x7f53a80b38c0 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5394003520 con 0x7f53a80b38c0 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730114200 0 0) 0x7f53a8141eb0 con 0x7f53a80b2800 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.383+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a80b22d0 con 0x7f53a80b2800 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741641762 0 0) 0x7f5394003520 con 0x7f53a80b38c0 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 >> v1:172.21.15.45:6790/0 conn(0x7f53a80a9c00 legacy=0x7f53a813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 >> v1:172.21.15.45:6789/0 conn(0x7f53a80b2800 legacy=0x7f53a8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a80aaf90 con 0x7f53a80b38c0 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a7fff640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:01.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5390001690 con 0x7f53a80b38c0 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: dump: 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:01.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53900018b0 con 0x7f53a80b38c0 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.384+0000 7f53a7fff640 4 set_mon_vals no callback set 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a5ffb640 1 -- 172.21.15.45:0/1021433381 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5390001bc0 con 0x7f53a80b38c0 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a5ffb640 10 monclient: dump: 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:01.371 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 10 monclient: shutdown 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 1 -- 172.21.15.45:0/1021433381 >> v1:172.21.15.116:6789/0 conn(0x7f53a80b38c0 legacy=0x7f53a814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 1 -- 172.21.15.45:0/1021433381 shutdown_connections 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 1 -- 172.21.15.45:0/1021433381 >> 172.21.15.45:0/1021433381 conn(0x7f53a8066a00 msgr2=0x7f53a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 1 -- 172.21.15.45:0/1021433381 shutdown_connections 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.385+0000 7f53a7fff640 1 -- 172.21.15.45:0/1021433381 wait complete. 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.386+0000 7f53a7fff640 10 monclient: build_initial_monmap 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.386+0000 7f53a7fff640 10 monclient: monmap: 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:01.372 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 1 librados: starting msgr at 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 1 librados: starting objecter 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 1 Processor -- start 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 1 -- start start 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 1 librados: setting wanted keys 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 1 librados: calling monclient init 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.387+0000 7f53a7fff640 10 monclient: init 2024-03-29T11:01:01.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.388+0000 7f53a7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:01.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.388+0000 7f53a7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f53a80a9c00 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f53a80b2800 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f53a80b38c0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a81d6fb0 con 0x7f53a80b38c0 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a80acd10 con 0x7f53a80a9c00 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a813ee00 con 0x7f53a80b2800 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.391+0000 7f53a7fff640 10 monclient(hunting): authenticate will time out at 1335.660972s 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.394+0000 7f53a6ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f53a80a9c00 0x7f53a81d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:37757/0 (socket says 172.21.15.45:37757) 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.394+0000 7f53a6ffd640 1 -- 172.21.15.45:0/3569181982 learned_addr learned my addr 172.21.15.45:0/3569181982 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 client.?.objecter ms_handle_connect 0x7f53a80b2800 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 client.?.objecter ms_handle_connect 0x7f53a80b38c0 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761436735 0 0) 0x7f53a81d6fb0 con 0x7f53a80b38c0 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5388004eb0 con 0x7f53a80b38c0 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 client.?.objecter ms_handle_connect 0x7f53a80a9c00 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972940685 0 0) 0x7f53a80acd10 con 0x7f53a80a9c00 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:01:01.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.395+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a81d6fb0 con 0x7f53a80a9c00 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1781991458 0 0) 0x7f5388004eb0 con 0x7f53a80b38c0 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53a80acd10 con 0x7f53a80b38c0 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5390001550 con 0x7f53a80b38c0 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 10 monclient(hunting): dump: 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:01.381 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.396+0000 7f539f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.397+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877459821 0 0) 0x7f53a81d6fb0 con 0x7f53a80a9c00 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5388004eb0 con 0x7f53a80a9c00 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212971107 0 0) 0x7f53a80acd10 con 0x7f53a80b38c0 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 >> v1:172.21.15.45:6790/0 conn(0x7f53a80b2800 legacy=0x7f53a8137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 >> v1:172.21.15.116:6789/0 conn(0x7f53a80a9c00 legacy=0x7f53a81d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 monclient: found mon.a 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a813c6e0 con 0x7f53a80b38c0 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:01.382 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53900016b0 con 0x7f53a80b38c0 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f5390003a40 con 0x7f53a80b38c0 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f539f7fe640 10 monclient: dump: 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 10 monclient: _renew_subs 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a81d7bc0 con 0x7f53a80b38c0 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 10 monclient: _renew_subs 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a8138730 con 0x7f53a80b38c0 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.398+0000 7f53a7fff640 1 librados: init done 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.400+0000 7f53a57fa640 4 set_mon_vals no callback set 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.400+0000 7f53a7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.400+0000 7f53a7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.400+0000 7f53a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.400+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538c005080 con 0x7f53a80b38c0 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.402+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f5390019a10 con 0x7f53a80b38c0 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.402+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f5390083750 con 0x7f53a80b38c0 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.402+0000 7f539f7fe640 10 client.4521.objecter ms_dispatch 0x7f53a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.402+0000 7f539f7fe640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.402+0000 7f539f7fe640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.402+0000 7f539f7fe640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.403+0000 7f53a6ffd640 10 monclient: get_auth_request con 0x7f538806ddb0 auth_method 0 2024-03-29T11:01:01.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.403+0000 7f539f7fe640 10 client.4521.objecter ms_handle_connect 0x7f538806ddb0 2024-03-29T11:01:01.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.413+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f539004f030 con 0x7f53a80b38c0 2024-03-29T11:01:01.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.413+0000 7f539f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.413+0000 7f539f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:01.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.752+0000 7f53a7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.752+0000 7f53a7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.752+0000 7f53a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:01.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.752+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f538c005330 con 0x7f53a80b38c0 2024-03-29T11:01:01.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.753+0000 7f539f7fe640 1 -- 172.21.15.45:0/3569181982 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f5390050f60 con 0x7f53a80b38c0 2024-03-29T11:01:01.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.753+0000 7f539f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.753+0000 7f539f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:01.734 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:01.734 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 10 librados: watch_flush enter 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 10 librados: watch_flush exit 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 10 client.4521.objecter successfully canceled tick 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 >> v1:172.21.15.116:6816/3290579615 conn(0x7f538806ddb0 legacy=0x7f5388070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 10 monclient: shutdown 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.757+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 >> v1:172.21.15.45:6789/0 conn(0x7f53a80b38c0 legacy=0x7f53a80ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.758+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 shutdown_connections 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.758+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 >> 172.21.15.45:0/3569181982 conn(0x7f53a8066a00 msgr2=0x7f53a813bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:01.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.758+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 shutdown_connections 2024-03-29T11:01:01.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.759+0000 7f53a7fff640 1 -- 172.21.15.45:0/3569181982 wait complete. 2024-03-29T11:01:01.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:01.759+0000 7f53a7fff640 1 librados: shutdown 2024-03-29T11:01:01.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:08.567 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:08.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.740+0000 7fa81a12c640 10 monclient: get_monmap_and_config 2024-03-29T11:01:08.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.740+0000 7fa81a12c640 10 monclient: build_initial_monmap 2024-03-29T11:01:08.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.740+0000 7fa81a12c640 10 monclient: monmap: 2024-03-29T11:01:08.720 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:08.720 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:08.720 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:08.741313+0000 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:08.741313+0000 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.741+0000 7fa81a12c640 1 Processor -- start 2024-03-29T11:01:08.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.742+0000 7fa81a12c640 1 -- start start 2024-03-29T11:01:08.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.742+0000 7fa81a12c640 10 monclient: init 2024-03-29T11:01:08.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient(hunting): picked mon.noname-a con 0x7fa8140b2800 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient(hunting): picked mon.noname-b con 0x7fa814142b20 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient(hunting): picked mon.noname-c con 0x7fa81413c040 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8140aad10 con 0x7fa8140b2800 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa814141f30 con 0x7fa81413c040 2024-03-29T11:01:08.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8140aaf60 con 0x7fa814142b20 2024-03-29T11:01:08.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:08.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.743+0000 7fa81a12c640 10 monclient(hunting): authenticate will time out at 1343.012105s 2024-03-29T11:01:08.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.745+0000 7fa81912a640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa8140b2800 0x7fa814142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54511/0 (socket says 172.21.15.45:54511) 2024-03-29T11:01:08.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.745+0000 7fa81912a640 1 -- 172.21.15.45:0/2879290474 learned_addr learned my addr 172.21.15.45:0/2879290474 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:08.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.746+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764333163 0 0) 0x7fa8140aad10 con 0x7fa8140b2800 2024-03-29T11:01:08.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.746+0000 7fa80bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.746+0000 7fa80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.746+0000 7fa80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.746+0000 7fa80bfff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:01:08.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.746+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800003520 con 0x7fa8140b2800 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2826175577 0 0) 0x7fa800003520 con 0x7fa8140b2800 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 >> v1:172.21.15.45:6790/0 conn(0x7fa814142b20 legacy=0x7fa81413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 >> v1:172.21.15.116:6789/0 conn(0x7fa81413c040 legacy=0x7fa81414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 monclient: found mon.noname-a 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8140b3900 con 0x7fa8140b2800 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: _finish_auth 0 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa810001690 con 0x7fa8140b2800 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: dump: 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa81a12c640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa81a12c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8100017f0 con 0x7fa8140b2800 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa80bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa81a12c640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:08.729 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.747+0000 7fa81a12c640 4 set_mon_vals no callback set 2024-03-29T11:01:08.730 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa81a12c640 10 monclient: shutdown 2024-03-29T11:01:08.730 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa81a12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa81a12c640 1 -- 172.21.15.45:0/2879290474 >> v1:172.21.15.45:6789/0 conn(0x7fa8140b2800 legacy=0x7fa814142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa80bfff640 1 -- 172.21.15.45:0/2879290474 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa810001c40 con 0x7fa8140b2800 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa80bfff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa81a12c640 1 -- 172.21.15.45:0/2879290474 shutdown_connections 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa81a12c640 1 -- 172.21.15.45:0/2879290474 >> 172.21.15.45:0/2879290474 conn(0x7fa81406d0e0 msgr2=0x7fa814066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.748+0000 7fa81a12c640 1 -- 172.21.15.45:0/2879290474 shutdown_connections 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.749+0000 7fa81a12c640 1 -- 172.21.15.45:0/2879290474 wait complete. 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.749+0000 7fa81a12c640 10 monclient: build_initial_monmap 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.750+0000 7fa81a12c640 10 monclient: monmap: 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:08.731 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.750+0000 7fa81a12c640 1 librados: starting msgr at 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.750+0000 7fa81a12c640 1 librados: starting objecter 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.750+0000 7fa81a12c640 1 Processor -- start 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 1 -- start start 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 1 librados: setting wanted keys 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 1 librados: calling monclient init 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 10 monclient: init 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.751+0000 7fa81a12c640 10 monclient(hunting): picked mon.noname-a con 0x7fa8140b2800 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 10 monclient(hunting): picked mon.noname-b con 0x7fa81413c040 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 10 monclient(hunting): picked mon.noname-c con 0x7fa814142b20 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa814145b50 con 0x7fa8140b2800 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81413f410 con 0x7fa81413c040 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa814145670 con 0x7fa814142b20 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:08.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa81a12c640 10 monclient(hunting): authenticate will time out at 1343.021527s 2024-03-29T11:01:08.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa818929640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa81413c040 0x7fa8141da610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:34050/0 (socket says 172.21.15.45:34050) 2024-03-29T11:01:08.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa818929640 1 -- 172.21.15.45:0/3696647076 learned_addr learned my addr 172.21.15.45:0/3696647076 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:08.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa809ffb640 10 client.?.objecter ms_handle_connect 0x7fa81413c040 2024-03-29T11:01:08.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.752+0000 7fa809ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.754+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167697741 0 0) 0x7fa81413f410 con 0x7fa81413c040 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.754+0000 7fa809ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.754+0000 7fa809ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.754+0000 7fa809ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.754+0000 7fa809ffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.754+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f8004eb0 con 0x7fa81413c040 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1572776787 0 0) 0x7fa7f8004eb0 con 0x7fa81413c040 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa81413f410 con 0x7fa81413c040 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa804001940 con 0x7fa81413c040 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 client.?.objecter ms_handle_connect 0x7fa8140b2800 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912448260 0 0) 0x7fa814145b50 con 0x7fa8140b2800 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.755+0000 7fa809ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f8004eb0 con 0x7fa8140b2800 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 10 client.?.objecter ms_handle_connect 0x7fa814142b20 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820607047 0 0) 0x7fa81413f410 con 0x7fa81413c040 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 >> v1:172.21.15.45:6790/0 conn(0x7fa814142b20 legacy=0x7fa8141e29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 >> v1:172.21.15.45:6789/0 conn(0x7fa8140b2800 legacy=0x7fa814136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 1 monclient: found mon.b 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa814145800 con 0x7fa81413c040 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa81a12c640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.756+0000 7fa81a12c640 10 monclient: _renew_subs 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa814145140 con 0x7fa81413c040 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 10 monclient: _renew_subs 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81413e810 con 0x7fa81413c040 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.757+0000 7fa81a12c640 1 librados: init done 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa804001d70 con 0x7fa81413c040 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa809ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fa8040038c0 con 0x7fa81413c040 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa809ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa809ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa809ffb640 10 monclient: dump: 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.758+0000 7fa80b7fe640 4 set_mon_vals no callback set 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fa804003a20 con 0x7fa81413c040 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa81a12c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa81a12c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa81a12c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f4005080 con 0x7fa81413c040 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fa804084a40 con 0x7fa81413c040 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa809ffb640 10 client.4546.objecter ms_dispatch 0x7fa814000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:08.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa809ffb640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:08.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa809ffb640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:08.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.761+0000 7fa809ffb640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:01:08.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.762+0000 7fa81912a640 10 monclient: get_auth_request con 0x7fa7f806dab0 auth_method 0 2024-03-29T11:01:08.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.763+0000 7fa809ffb640 10 client.4546.objecter ms_handle_connect 0x7fa7f806dab0 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.770+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa804050320 con 0x7fa81413c040 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.770+0000 7fa809ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:08.770+0000 7fa809ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:09.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.092+0000 7fa81a12c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.092+0000 7fa81a12c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.092+0000 7fa81a12c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:09.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.092+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7f4005330 con 0x7fa81413c040 2024-03-29T11:01:09.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.098+0000 7fa809ffb640 1 -- 172.21.15.45:0/3696647076 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fa804052250 con 0x7fa81413c040 2024-03-29T11:01:09.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.098+0000 7fa809ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.098+0000 7fa809ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:09.079 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:09.079 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:09.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.102+0000 7fa81a12c640 10 librados: watch_flush enter 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.105+0000 7fa81a12c640 10 librados: watch_flush exit 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.105+0000 7fa81a12c640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.105+0000 7fa81a12c640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.105+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 >> v1:172.21.15.116:6816/3290579615 conn(0x7fa7f806dab0 legacy=0x7fa7f806ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.106+0000 7fa81a12c640 10 monclient: shutdown 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.106+0000 7fa81a12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.106+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 >> v1:172.21.15.116:6789/0 conn(0x7fa81413c040 legacy=0x7fa8141da610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.106+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 shutdown_connections 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.106+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 >> 172.21.15.45:0/3696647076 conn(0x7fa81406d0e0 msgr2=0x7fa8140b8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.107+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 shutdown_connections 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.107+0000 7fa81a12c640 1 -- 172.21.15.45:0/3696647076 wait complete. 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:09.107+0000 7fa81a12c640 1 librados: shutdown 2024-03-29T11:01:09.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:15.913 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:16.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.070+0000 7f2c4a5bc640 10 monclient: get_monmap_and_config 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.070+0000 7f2c4a5bc640 10 monclient: build_initial_monmap 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.070+0000 7f2c4a5bc640 10 monclient: monmap: 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:16.070868+0000 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:16.070868+0000 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:16.050 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:16.051 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.071+0000 7f2c4a5bc640 1 Processor -- start 2024-03-29T11:01:16.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.071+0000 7f2c4a5bc640 1 -- start start 2024-03-29T11:01:16.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.071+0000 7f2c4a5bc640 10 monclient: init 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f2c441422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f2c44142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f2c440b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c44142c80 con 0x7f2c441422b0 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440aae40 con 0x7f2c440b17d0 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4413fbd0 con 0x7f2c44142690 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:16.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.072+0000 7f2c4a5bc640 10 monclient(hunting): authenticate will time out at 1350.341430s 2024-03-29T11:01:16.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.073+0000 7f2c437fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2c44142690 0x7f2c440b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55494/0 (socket says 172.21.15.45:55494) 2024-03-29T11:01:16.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.073+0000 7f2c437fe640 1 -- 172.21.15.45:0/669694264 learned_addr learned my addr 172.21.15.45:0/669694264 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:16.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.073+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685452180 0 0) 0x7f2c4413fbd0 con 0x7f2c44142690 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2c003520 con 0x7f2c44142690 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906791101 0 0) 0x7f2c440aae40 con 0x7f2c440b17d0 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:01:16.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c4413fbd0 con 0x7f2c440b17d0 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1686313936 0 0) 0x7f2c2c003520 con 0x7f2c44142690 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.074+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 >> v1:172.21.15.116:6789/0 conn(0x7f2c440b17d0 legacy=0x7f2c440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 >> v1:172.21.15.45:6789/0 conn(0x7f2c441422b0 legacy=0x7f2c4413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 1 monclient: found mon.noname-b 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c44151940 con 0x7f2c44142690 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2c38001690 con 0x7f2c44142690 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: dump: 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c4a5bc640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c4a5bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c380017f0 con 0x7f2c44142690 2024-03-29T11:01:16.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 1 -- 172.21.15.45:0/669694264 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2c38001b80 con 0x7f2c44142690 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c42ffd640 10 monclient: dump: 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.075+0000 7f2c4a5bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.076+0000 7f2c4a5bc640 4 set_mon_vals no callback set 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.076+0000 7f2c4a5bc640 10 monclient: shutdown 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.076+0000 7f2c4a5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.076+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/669694264 >> v1:172.21.15.45:6790/0 conn(0x7f2c44142690 legacy=0x7f2c440b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.077+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/669694264 shutdown_connections 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.077+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/669694264 >> 172.21.15.45:0/669694264 conn(0x7f2c44066a00 msgr2=0x7f2c44066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.077+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/669694264 shutdown_connections 2024-03-29T11:01:16.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.077+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/669694264 wait complete. 2024-03-29T11:01:16.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 10 monclient: build_initial_monmap 2024-03-29T11:01:16.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 10 monclient: monmap: 2024-03-29T11:01:16.066 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:16.066 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:16.066 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 1 librados: starting msgr at 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 1 librados: starting objecter 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 1 Processor -- start 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 1 -- start start 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 1 librados: setting wanted keys 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 1 librados: calling monclient init 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.078+0000 7f2c4a5bc640 10 monclient: init 2024-03-29T11:01:16.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.079+0000 7f2c4a5bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.079+0000 7f2c4a5bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.079+0000 7f2c4a5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f2c440b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f2c441422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f2c44142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4412e070 con 0x7f2c441422b0 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4412e350 con 0x7f2c440b17d0 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4412f160 con 0x7f2c44142690 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c4a5bc640 10 monclient(hunting): authenticate will time out at 1350.350312s 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c43fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2c440b17d0 0x7f2c44150520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:48420/0 (socket says 172.21.15.45:48420) 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c43fff640 1 -- 172.21.15.45:0/1691800101 learned_addr learned my addr 172.21.15.45:0/1691800101 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 10 client.?.objecter ms_handle_connect 0x7f2c440b17d0 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:16.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352036507 0 0) 0x7f2c4412e350 con 0x7f2c440b17d0 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.081+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c30004eb0 con 0x7f2c440b17d0 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 client.?.objecter ms_handle_connect 0x7f2c44142690 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2205287954 0 0) 0x7f2c30004eb0 con 0x7f2c440b17d0 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c4412e350 con 0x7f2c440b17d0 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2c3c001d50 con 0x7f2c440b17d0 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:16.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284087750 0 0) 0x7f2c4412f160 con 0x7f2c44142690 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c30004eb0 con 0x7f2c44142690 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 552517100 0 0) 0x7f2c30004eb0 con 0x7f2c44142690 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c4412f160 con 0x7f2c44142690 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2c34001930 con 0x7f2c44142690 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.082+0000 7f2c40ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4237417113 0 0) 0x7f2c4412f160 con 0x7f2c44142690 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 >> v1:172.21.15.116:6789/0 conn(0x7f2c440b17d0 legacy=0x7f2c44150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 >> v1:172.21.15.45:6789/0 conn(0x7f2c441422b0 legacy=0x7f2c4412ea50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 1 monclient: found mon.c 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4412f2f0 con 0x7f2c44142690 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 10 monclient: _finish_auth 0 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 10 monclient: _renew_subs 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c440b7650 con 0x7f2c44142690 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 10 monclient: _renew_subs 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4412d4e0 con 0x7f2c44142690 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.083+0000 7f2c4a5bc640 1 librados: init done 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c34001d40 con 0x7f2c44142690 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c427fc640 4 set_mon_vals no callback set 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f2c34003790 con 0x7f2c44142690 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.084+0000 7f2c40ff9640 10 monclient: dump: 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.074 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.085+0000 7f2c4a5bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.085+0000 7f2c4a5bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.085+0000 7f2c4a5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.085+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c24005080 con 0x7f2c44142690 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.086+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f2c340197e0 con 0x7f2c44142690 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.087+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f2c34083660 con 0x7f2c44142690 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.087+0000 7f2c40ff9640 10 client.4589.objecter ms_dispatch 0x7f2c44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.087+0000 7f2c40ff9640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.087+0000 7f2c40ff9640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.087+0000 7f2c40ff9640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:01:16.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.088+0000 7f2c43fff640 10 monclient: get_auth_request con 0x7f2c3006e060 auth_method 0 2024-03-29T11:01:16.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.090+0000 7f2c40ff9640 10 client.4589.objecter ms_handle_connect 0x7f2c3006e060 2024-03-29T11:01:16.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.094+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c3404ed70 con 0x7f2c44142690 2024-03-29T11:01:16.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.094+0000 7f2c40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.094+0000 7f2c40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:16.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.422+0000 7f2c4a5bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.422+0000 7f2c4a5bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.422+0000 7f2c4a5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:16.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.422+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c24005330 con 0x7f2c44142690 2024-03-29T11:01:16.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.424+0000 7f2c40ff9640 1 -- 172.21.15.45:0/1691800101 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f2c3404ed70 con 0x7f2c44142690 2024-03-29T11:01:16.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.424+0000 7f2c40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.424+0000 7f2c40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:16.404 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:16.405 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:16.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 10 librados: watch_flush enter 2024-03-29T11:01:16.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 10 librados: watch_flush exit 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 >> v1:172.21.15.116:6816/3290579615 conn(0x7f2c3006e060 legacy=0x7f2c30070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 10 monclient: shutdown 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 >> v1:172.21.15.45:6790/0 conn(0x7f2c44142690 legacy=0x7f2c441e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 shutdown_connections 2024-03-29T11:01:16.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.429+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 >> 172.21.15.45:0/1691800101 conn(0x7f2c44066a00 msgr2=0x7f2c440b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:16.410 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.430+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 shutdown_connections 2024-03-29T11:01:16.410 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.430+0000 7f2c4a5bc640 1 -- 172.21.15.45:0/1691800101 wait complete. 2024-03-29T11:01:16.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:16.431+0000 7f2c4a5bc640 1 librados: shutdown 2024-03-29T11:01:16.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:23.237 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:23.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.411+0000 7f30b34fb640 10 monclient: get_monmap_and_config 2024-03-29T11:01:23.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.411+0000 7f30b34fb640 10 monclient: build_initial_monmap 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.411+0000 7f30b34fb640 10 monclient: monmap: 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:23.412556+0000 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:23.412556+0000 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:23.393 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.414+0000 7f30b34fb640 1 Processor -- start 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.414+0000 7f30b34fb640 1 -- start start 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.414+0000 7f30b34fb640 10 monclient: init 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient(hunting): picked mon.noname-c con 0x7f30ac0a9ac0 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient(hunting): picked mon.noname-b con 0x7f30ac13ee90 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient(hunting): picked mon.noname-a con 0x7f30ac0af900 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0afd10 con 0x7f30ac0af900 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0af6f0 con 0x7f30ac0a9ac0 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0aac80 con 0x7f30ac13ee90 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.415+0000 7f30b34fb640 10 monclient(hunting): authenticate will time out at 1357.684665s 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.419+0000 7f30b0a96640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f30ac13ee90 0x7f30ac13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55713/0 (socket says 172.21.15.45:55713) 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.419+0000 7f30b0a96640 1 -- 172.21.15.45:0/461411567 learned_addr learned my addr 172.21.15.45:0/461411567 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044354235 0 0) 0x7f30ac0afd10 con 0x7f30ac0af900 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003520 con 0x7f30ac0af900 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190790794 0 0) 0x7f30ac0aac80 con 0x7f30ac13ee90 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.422+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac0afd10 con 0x7f30ac13ee90 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636878664 0 0) 0x7f3098003520 con 0x7f30ac0af900 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 >> v1:172.21.15.45:6790/0 conn(0x7f30ac13ee90 legacy=0x7f30ac13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 >> v1:172.21.15.116:6789/0 conn(0x7f30ac0a9ac0 legacy=0x7f30ac0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 monclient: found mon.noname-a 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0aaf60 con 0x7f30ac0af900 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f309c001690 con 0x7f30ac0af900 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: dump: 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f309c0017f0 con 0x7f30ac0af900 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 1 -- 172.21.15.45:0/461411567 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f309c001c00 con 0x7f30ac0af900 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.424+0000 7f30a3fff640 10 monclient: dump: 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.426+0000 7f30b34fb640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.426+0000 7f30b34fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.426+0000 7f30b34fb640 4 set_mon_vals no callback set 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.426+0000 7f30b34fb640 10 monclient: shutdown 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.426+0000 7f30b34fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.426+0000 7f30b34fb640 1 -- 172.21.15.45:0/461411567 >> v1:172.21.15.45:6789/0 conn(0x7f30ac0af900 legacy=0x7f30ac149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.427+0000 7f30b34fb640 1 -- 172.21.15.45:0/461411567 shutdown_connections 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.427+0000 7f30b34fb640 1 -- 172.21.15.45:0/461411567 >> 172.21.15.45:0/461411567 conn(0x7f30ac0b35e0 msgr2=0x7f30ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.428+0000 7f30b34fb640 1 -- 172.21.15.45:0/461411567 shutdown_connections 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.429+0000 7f30b34fb640 1 -- 172.21.15.45:0/461411567 wait complete. 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.429+0000 7f30b34fb640 10 monclient: build_initial_monmap 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.429+0000 7f30b34fb640 10 monclient: monmap: 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 1 librados: starting msgr at 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 1 librados: starting objecter 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 1 Processor -- start 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 1 -- start start 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 1 librados: setting wanted keys 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 1 librados: calling monclient init 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.430+0000 7f30b34fb640 10 monclient: init 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.431+0000 7f30b34fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.431+0000 7f30b34fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.431+0000 7f30b34fb640 10 monclient(hunting): picked mon.noname-a con 0x7f30ac0a9ac0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.431+0000 7f30b34fb640 10 monclient(hunting): picked mon.noname-c con 0x7f30ac0af900 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.432+0000 7f30b34fb640 10 monclient(hunting): picked mon.noname-b con 0x7f30ac13ee90 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.432+0000 7f30b34fb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac13a1e0 con 0x7f30ac0a9ac0 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.432+0000 7f30b34fb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac13b960 con 0x7f30ac13ee90 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.432+0000 7f30b34fb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac138c40 con 0x7f30ac0af900 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.432+0000 7f30b34fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.432+0000 7f30b34fb640 10 monclient(hunting): authenticate will time out at 1357.701154s 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.435+0000 7f30b0a96640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f30ac0af900 0x7f30ac1e4470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55803/0 (socket says 172.21.15.45:55803) 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.435+0000 7f30b0a96640 1 -- 172.21.15.45:0/2655018897 learned_addr learned my addr 172.21.15.45:0/2655018897 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.435+0000 7f30a1ffb640 10 client.?.objecter ms_handle_connect 0x7f30ac0af900 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.435+0000 7f30a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 10 client.?.objecter ms_handle_connect 0x7f30ac13ee90 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631542790 0 0) 0x7f30ac13b960 con 0x7f30ac13ee90 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.436+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3090004eb0 con 0x7f30ac13ee90 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455616968 0 0) 0x7f3090004eb0 con 0x7f30ac13ee90 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30ac13b960 con 0x7f30ac13ee90 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f309c001c00 con 0x7f30ac13ee90 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.437+0000 7f30a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1922906887 0 0) 0x7f30ac13b960 con 0x7f30ac13ee90 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 >> v1:172.21.15.45:6790/0 conn(0x7f30ac0af900 legacy=0x7f30ac1e4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 >> v1:172.21.15.45:6789/0 conn(0x7f30ac0a9ac0 legacy=0x7f30ac139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 1 monclient: found mon.b 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac138dd0 con 0x7f30ac13ee90 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 10 monclient: _renew_subs 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac139cc0 con 0x7f30ac13ee90 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 10 monclient: _renew_subs 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac139fc0 con 0x7f30ac13ee90 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.438+0000 7f30b34fb640 1 librados: init done 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f309c0038a0 con 0x7f30ac13ee90 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a37fe640 4 set_mon_vals no callback set 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f309c003a10 con 0x7f30ac13ee90 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.440+0000 7f30a1ffb640 10 monclient: dump: 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.441+0000 7f30b34fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.441+0000 7f30b34fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.441+0000 7f30b34fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.441+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c005080 con 0x7f30ac13ee90 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.442+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f309c0199e0 con 0x7f30ac13ee90 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.442+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f309c083880 con 0x7f30ac13ee90 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.442+0000 7f30a1ffb640 10 client.4555.objecter ms_dispatch 0x7f30ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.442+0000 7f30a1ffb640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.442+0000 7f30a1ffb640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.443+0000 7f30a1ffb640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.444+0000 7f30b1297640 10 monclient: get_auth_request con 0x7f309006dc20 auth_method 0 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.445+0000 7f30a1ffb640 10 client.4555.objecter ms_handle_connect 0x7f309006dc20 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.449+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f309c04ef90 con 0x7f30ac13ee90 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.449+0000 7f30a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.449+0000 7f30a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:23.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.863+0000 7f30b34fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.863+0000 7f30b34fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.863+0000 7f30b34fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:01:23.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.863+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f308c005330 con 0x7f30ac13ee90 2024-03-29T11:01:23.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.864+0000 7f30a1ffb640 1 -- 172.21.15.45:0/2655018897 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f309c051090 con 0x7f30ac13ee90 2024-03-29T11:01:23.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.864+0000 7f30a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.864+0000 7f30a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:23.848 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:23.848 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.868+0000 7f30b34fb640 10 librados: watch_flush enter 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 10 librados: watch_flush exit 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 10 client.4555.objecter successfully canceled tick 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 >> v1:172.21.15.116:6816/3290579615 conn(0x7f309006dc20 legacy=0x7f30900700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 10 monclient: shutdown 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.849 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 >> v1:172.21.15.116:6789/0 conn(0x7f30ac13ee90 legacy=0x7f30ac1e4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 shutdown_connections 2024-03-29T11:01:23.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 >> 172.21.15.45:0/2655018897 conn(0x7f30ac0b35e0 msgr2=0x7f30ac0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 shutdown_connections 2024-03-29T11:01:23.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 -- 172.21.15.45:0/2655018897 wait complete. 2024-03-29T11:01:23.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:23.869+0000 7f30b34fb640 1 librados: shutdown 2024-03-29T11:01:23.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:30.671 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.844+0000 7f0fcdda3640 10 monclient: get_monmap_and_config 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.844+0000 7f0fcdda3640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.844+0000 7f0fcdda3640 10 monclient: monmap: 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:30.845255+0000 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:30.845255+0000 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:30.824 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:30.825 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:30.825 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:30.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.845+0000 7f0fcdda3640 1 Processor -- start 2024-03-29T11:01:30.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.845+0000 7f0fcdda3640 1 -- start start 2024-03-29T11:01:30.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.845+0000 7f0fcdda3640 10 monclient: init 2024-03-29T11:01:30.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:30.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc80b1f60 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc8071eb0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc8143d10 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc806ef80 con 0x7f0fc8071eb0 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc806f260 con 0x7f0fc80b1f60 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc80b40c0 con 0x7f0fc8143d10 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.846+0000 7f0fcdda3640 10 monclient(hunting): authenticate will time out at 1365.115478s 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.849+0000 7f0fc77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0fc80b1f60 0x7f0fc8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:49341/0 (socket says 172.21.15.45:49341) 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.849+0000 7f0fc77fe640 1 -- 172.21.15.45:0/2799439274 learned_addr learned my addr 172.21.15.45:0/2799439274 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068904493 0 0) 0x7f0fc806f260 con 0x7f0fc80b1f60 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb0003520 con 0x7f0fc80b1f60 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3099688227 0 0) 0x7f0fb0003520 con 0x7f0fc80b1f60 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.850+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 >> v1:172.21.15.45:6790/0 conn(0x7f0fc8143d10 legacy=0x7f0fc8146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 >> v1:172.21.15.45:6789/0 conn(0x7f0fc8071eb0 legacy=0x7f0fc8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 1 monclient: found mon.noname-c 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8155d70 con 0x7f0fc80b1f60 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fcdda3640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fcdda3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0fc0001a10 con 0x7f0fc80b1f60 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fc67fc640 10 monclient: dump: 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:30.835 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.851+0000 7f0fcdda3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.852+0000 7f0fc67fc640 1 -- 172.21.15.45:0/2799439274 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fc0001ca0 con 0x7f0fc80b1f60 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.852+0000 7f0fc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.852+0000 7f0fcdda3640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.852+0000 7f0fcdda3640 4 set_mon_vals no callback set 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.853+0000 7f0fcdda3640 10 monclient: shutdown 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.853+0000 7f0fcdda3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.853+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2799439274 >> v1:172.21.15.116:6789/0 conn(0x7f0fc80b1f60 legacy=0x7f0fc8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.853+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2799439274 shutdown_connections 2024-03-29T11:01:30.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.853+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2799439274 >> 172.21.15.45:0/2799439274 conn(0x7f0fc8066a00 msgr2=0x7f0fc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:30.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.854+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2799439274 shutdown_connections 2024-03-29T11:01:30.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.854+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2799439274 wait complete. 2024-03-29T11:01:30.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.855+0000 7f0fcdda3640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.855+0000 7f0fcdda3640 10 monclient: monmap: 2024-03-29T11:01:30.838 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:30.838 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.855+0000 7f0fcdda3640 1 librados: starting msgr at 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.855+0000 7f0fcdda3640 1 librados: starting objecter 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.855+0000 7f0fcdda3640 1 Processor -- start 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 1 -- start start 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 1 librados: setting wanted keys 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 1 librados: calling monclient init 2024-03-29T11:01:30.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient: init 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc80b1f60 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc8143d10 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc8138610 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc813c020 con 0x7f0fc80b1f60 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc80b7420 con 0x7f0fc8138610 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc813a1a0 con 0x7f0fc8143d10 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.856+0000 7f0fcdda3640 10 monclient(hunting): authenticate will time out at 1365.125960s 2024-03-29T11:01:30.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.860+0000 7f0fc77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0fc80b1f60 0x7f0fc813a6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42393/0 (socket says 172.21.15.45:42393) 2024-03-29T11:01:30.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.860+0000 7f0fc77fe640 1 -- 172.21.15.45:0/2806680930 learned_addr learned my addr 172.21.15.45:0/2806680930 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 10 client.?.objecter ms_handle_connect 0x7f0fc80b1f60 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347631074 0 0) 0x7f0fc813c020 con 0x7f0fc80b1f60 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:01:30.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.864+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb4004eb0 con 0x7f0fc80b1f60 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4167287100 0 0) 0x7f0fb4004eb0 con 0x7f0fc80b1f60 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fc813c020 con 0x7f0fc80b1f60 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0fc0004e60 con 0x7f0fc80b1f60 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 10 monclient(hunting): dump: 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:30.846 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3088389877 0 0) 0x7f0fc813c020 con 0x7f0fc80b1f60 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.865+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6790/0 conn(0x7f0fc8143d10 legacy=0x7f0fc80b7920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.116:6789/0 conn(0x7f0fc8138610 legacy=0x7f0fc80b8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 1 monclient: found mon.a 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc80b7040 con 0x7f0fc80b1f60 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0faffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 10 monclient: _renew_subs 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc813ade0 con 0x7f0fc80b1f60 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 10 monclient: _renew_subs 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc813f980 con 0x7f0fc80b1f60 2024-03-29T11:01:30.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.866+0000 7f0fcdda3640 1 librados: init done 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fc0001e90 con 0x7f0fc80b1f60 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0faffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0fc0005550 con 0x7f0fc80b1f60 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0faffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0faffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0faffff640 10 monclient: dump: 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:30.850 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.867+0000 7f0fc57fa640 4 set_mon_vals no callback set 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.868+0000 7f0fcdda3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.868+0000 7f0fcdda3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.868+0000 7f0fcdda3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.868+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa8005080 con 0x7f0fc80b1f60 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.869+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f0fc00056b0 con 0x7f0fc80b1f60 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.869+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f0fc0085280 con 0x7f0fc80b1f60 2024-03-29T11:01:30.851 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.869+0000 7f0faffff640 10 client.4539.objecter ms_dispatch 0x7f0fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:30.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.869+0000 7f0faffff640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:30.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.869+0000 7f0faffff640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:30.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.869+0000 7f0faffff640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:01:30.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.873+0000 7f0fc6ffd640 10 monclient: get_auth_request con 0x7f0fb406dc80 auth_method 0 2024-03-29T11:01:30.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.875+0000 7f0faffff640 10 client.4539.objecter ms_handle_connect 0x7f0fb406dc80 2024-03-29T11:01:30.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0fc77fe640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6789/0 conn(0x7f0fc80b1f60 legacy=0x7f0fc813a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:01:30.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0fc77fe640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6789/0 conn(0x7f0fc80b1f60 legacy=0x7f0fc813a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:01:30.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0fc77fe640 1 --1- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6789/0 conn(0x7f0fc80b1f60 0x7f0fc813a6d0 :-1 s=OPENED pgs=892955 cs=1 l=1).handle_message read tag failed 2024-03-29T11:01:30.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0fc77fe640 1 --1- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6789/0 conn(0x7f0fc80b1f60 0x7f0fc813a6d0 :-1 s=OPENED pgs=892955 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:01:30.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 10 monclient: ms_handle_reset current mon v1:172.21.15.45:6789/0 2024-03-29T11:01:30.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6789/0 conn(0x7f0fc80b1f60 legacy=0x7f0fc813a6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 10 monclient(hunting): picked mon.b con 0x7f0fb4072a00 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 10 monclient(hunting): picked mon.c con 0x7f0fc8143d10 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 10 monclient(hunting): picked mon.a con 0x7f0fb407a530 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0fc813c020 con 0x7f0fb407a530 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.877+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0fc80b7420 con 0x7f0fb4072a00 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.878+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0fc813a1a0 con 0x7f0fc8143d10 2024-03-29T11:01:30.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.878+0000 7f0faffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 client.4539.objecter ms_handle_connect 0x7f0fc8143d10 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 client.4539.objecter resend_mon_ops 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922551878 0 0) 0x7f0fc813a1a0 con 0x7f0fc8143d10 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f0fb4080670 con 0x7f0fc8143d10 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 client.4539.objecter ms_handle_connect 0x7f0fb4072a00 2024-03-29T11:01:30.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.885+0000 7f0faffff640 10 client.4539.objecter resend_mon_ops 2024-03-29T11:01:30.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3335257196 0 0) 0x7f0fb4080670 con 0x7f0fc8143d10 2024-03-29T11:01:30.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.116:6789/0 conn(0x7f0fb4072a00 legacy=0x7f0fb4074df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6789/0 conn(0x7f0fb407a530 legacy=0x7f0fb407c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 monclient: found mon.c 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f0fb4004eb0 con 0x7f0fc8143d10 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc813c020 con 0x7f0fc8143d10 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0fb8001480 con 0x7f0fc8143d10 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.887+0000 7f0faffff640 10 monclient: dump: 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:30.869 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:30.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.892+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fb8001220 con 0x7f0fc8143d10 2024-03-29T11:01:30.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.892+0000 7f0faffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.892+0000 7f0fc5ffb640 4 set_mon_vals no callback set 2024-03-29T11:01:30.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.892+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f0fb80016a0 con 0x7f0fc8143d10 2024-03-29T11:01:30.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.903+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fb8051cf0 con 0x7f0fc8143d10 2024-03-29T11:01:30.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.903+0000 7f0faffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:30.903+0000 7f0faffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:31.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.239+0000 7f0fcdda3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.239+0000 7f0fcdda3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.239+0000 7f0fcdda3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:31.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.239+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fa8005330 con 0x7f0fc8143d10 2024-03-29T11:01:31.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.241+0000 7f0faffff640 1 -- 172.21.15.45:0/2806680930 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f0fb8052a90 con 0x7f0fc8143d10 2024-03-29T11:01:31.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.241+0000 7f0faffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.241+0000 7f0faffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:31.223 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:31.223 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:31.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.246+0000 7f0fcdda3640 10 librados: watch_flush enter 2024-03-29T11:01:31.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.246+0000 7f0fcdda3640 10 librados: watch_flush exit 2024-03-29T11:01:31.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.246+0000 7f0fcdda3640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:01:31.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.246+0000 7f0fcdda3640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:01:31.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.250+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.116:6816/3290579615 conn(0x7f0fb406dc80 legacy=0x7f0fb4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:31.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.250+0000 7f0fcdda3640 10 monclient: shutdown 2024-03-29T11:01:31.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.250+0000 7f0fcdda3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:31.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.250+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 >> v1:172.21.15.45:6790/0 conn(0x7f0fc8143d10 legacy=0x7f0fb40788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:31.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.251+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 shutdown_connections 2024-03-29T11:01:31.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.251+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 >> 172.21.15.45:0/2806680930 conn(0x7f0fc8066a00 msgr2=0x7f0fc8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:31.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.251+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 shutdown_connections 2024-03-29T11:01:31.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.251+0000 7f0fcdda3640 1 -- 172.21.15.45:0/2806680930 wait complete. 2024-03-29T11:01:31.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:31.251+0000 7f0fcdda3640 1 librados: shutdown 2024-03-29T11:01:31.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:38.057 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:38.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.210+0000 7fe3627af640 10 monclient: get_monmap_and_config 2024-03-29T11:01:38.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.210+0000 7fe3627af640 10 monclient: build_initial_monmap 2024-03-29T11:01:38.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.210+0000 7fe3627af640 10 monclient: monmap: 2024-03-29T11:01:38.190 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:38.190 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:38.211285+0000 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:38.211285+0000 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:38.191 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:38.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.211+0000 7fe3627af640 1 Processor -- start 2024-03-29T11:01:38.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.211+0000 7fe3627af640 1 -- start start 2024-03-29T11:01:38.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.211+0000 7fe3627af640 10 monclient: init 2024-03-29T11:01:38.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.212+0000 7fe3627af640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:38.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.212+0000 7fe3627af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:38.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.212+0000 7fe3627af640 10 monclient(hunting): picked mon.noname-b con 0x7fe35c0b0860 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:38.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.212+0000 7fe3627af640 10 monclient(hunting): picked mon.noname-c con 0x7fe35c13ef30 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:38.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.212+0000 7fe35b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe35c13ef30 0x7fe35c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:54364/0 (socket says 172.21.15.45:54364) 2024-03-29T11:01:38.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.212+0000 7fe35bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe35c0b0860 0x7fe35c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60942/0 (socket says 172.21.15.45:60942) 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe3627af640 10 monclient(hunting): picked mon.noname-a con 0x7fe35c0af8e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe35b7fe640 1 -- 172.21.15.45:0/1824868463 learned_addr learned my addr 172.21.15.45:0/1824868463 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0b0c40 con 0x7fe35c0af8e0 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0aed10 con 0x7fe35c13ef30 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0afd50 con 0x7fe35c0b0860 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe3627af640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe3627af640 10 monclient(hunting): authenticate will time out at 1372.482263s 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439803800 0 0) 0x7fe35c0afd50 con 0x7fe35c0b0860 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe35affd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.213+0000 7fe35affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.214+0000 7fe35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.214+0000 7fe35affd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:01:38.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344003520 con 0x7fe35c0b0860 2024-03-29T11:01:38.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067935943 0 0) 0x7fe35c0aed10 con 0x7fe35c13ef30 2024-03-29T11:01:38.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe35c0afd50 con 0x7fe35c13ef30 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051057253 0 0) 0x7fe344003520 con 0x7fe35c0b0860 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 >> v1:172.21.15.116:6789/0 conn(0x7fe35c13ef30 legacy=0x7fe35c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 >> v1:172.21.15.45:6789/0 conn(0x7fe35c0af8e0 legacy=0x7fe35c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.215+0000 7fe35affd640 1 monclient: found mon.noname-b 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0ab1b0 con 0x7fe35c0b0860 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 10 monclient: _finish_auth 0 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:38.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe3627af640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe3627af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe34c000ed0 con 0x7fe35c0b0860 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.216+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 10 monclient: dump: 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:38.200 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe34c0010f0 con 0x7fe35c0b0860 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe3627af640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 1 -- 172.21.15.45:0/1824868463 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe34c001480 con 0x7fe35c0b0860 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe3627af640 4 set_mon_vals no callback set 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.217+0000 7fe35affd640 10 monclient: dump: 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.201 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.218+0000 7fe3627af640 10 monclient: shutdown 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.218+0000 7fe3627af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.218+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 >> v1:172.21.15.45:6790/0 conn(0x7fe35c0b0860 legacy=0x7fe35c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.218+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 shutdown_connections 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.218+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 >> 172.21.15.45:0/1824868463 conn(0x7fe35c06b660 msgr2=0x7fe35c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.219+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 shutdown_connections 2024-03-29T11:01:38.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.219+0000 7fe3627af640 1 -- 172.21.15.45:0/1824868463 wait complete. 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 10 monclient: build_initial_monmap 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 10 monclient: monmap: 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:38.209 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 1 librados: starting msgr at 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 1 librados: starting objecter 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 1 Processor -- start 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 1 -- start start 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 1 librados: setting wanted keys 2024-03-29T11:01:38.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 1 librados: calling monclient init 2024-03-29T11:01:38.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.220+0000 7fe3627af640 10 monclient: init 2024-03-29T11:01:38.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:38.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient(hunting): picked mon.noname-b con 0x7fe35c0af8e0 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient(hunting): picked mon.noname-a con 0x7fe35c0b0860 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient(hunting): picked mon.noname-c con 0x7fe35c13ef30 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0717f0 con 0x7fe35c0b0860 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0709e0 con 0x7fe35c0af8e0 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c070cc0 con 0x7fe35c13ef30 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe3627af640 10 monclient(hunting): authenticate will time out at 1372.490667s 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe360d4c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe35c13ef30 0x7fe35c06ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34540/0 (socket says 172.21.15.45:34540) 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.221+0000 7fe360d4c640 1 -- 172.21.15.45:0/2942472575 learned_addr learned my addr 172.21.15.45:0/2942472575 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 10 client.?.objecter ms_handle_connect 0x7fe35c13ef30 2024-03-29T11:01:38.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948217279 0 0) 0x7fe35c070cc0 con 0x7fe35c13ef30 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.222+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe348004eb0 con 0x7fe35c13ef30 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451523298 0 0) 0x7fe348004eb0 con 0x7fe35c13ef30 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe35c070cc0 con 0x7fe35c13ef30 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe354001cd0 con 0x7fe35c13ef30 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:38.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:38.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 10 client.?.objecter ms_handle_connect 0x7fe35c0af8e0 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3992866004 0 0) 0x7fe35c070cc0 con 0x7fe35c13ef30 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 >> v1:172.21.15.116:6789/0 conn(0x7fe35c0af8e0 legacy=0x7fe35c0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.223+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 >> v1:172.21.15.45:6789/0 conn(0x7fe35c0b0860 legacy=0x7fe35c06e660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 1 monclient: found mon.c 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c135010 con 0x7fe35c13ef30 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 10 monclient: _finish_auth 0 2024-03-29T11:01:38.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 10 monclient: _renew_subs 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c0b3da0 con 0x7fe35c13ef30 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 10 monclient: _renew_subs 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c14f080 con 0x7fe35c13ef30 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.224+0000 7fe3627af640 1 librados: init done 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.226+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe354001fd0 con 0x7fe35c13ef30 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.226+0000 7fe358ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.226+0000 7fe35a7fc640 4 set_mon_vals no callback set 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe354003de0 con 0x7fe35c13ef30 2024-03-29T11:01:38.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe358ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe358ff9640 10 monclient: dump: 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe354004000 con 0x7fe35c13ef30 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe3627af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe3627af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe3627af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.227+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c005080 con 0x7fe35c13ef30 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe35bfff640 10 monclient: get_auth_request con 0x7fe34806dbe0 auth_method 0 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7fe354084df0 con 0x7fe35c13ef30 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe358ff9640 10 client.4601.objecter ms_dispatch 0x7fe35c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe358ff9640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe358ff9640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe358ff9640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:01:38.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.228+0000 7fe358ff9640 10 client.4601.objecter ms_handle_connect 0x7fe34806dbe0 2024-03-29T11:01:38.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.234+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe354050790 con 0x7fe35c13ef30 2024-03-29T11:01:38.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.234+0000 7fe358ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.234+0000 7fe358ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:38.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.555+0000 7fe3627af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.555+0000 7fe3627af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.555+0000 7fe3627af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:38.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.555+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe33c005330 con 0x7fe35c13ef30 2024-03-29T11:01:38.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.558+0000 7fe358ff9640 1 -- 172.21.15.45:0/2942472575 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7fe3540526c0 con 0x7fe35c13ef30 2024-03-29T11:01:38.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.558+0000 7fe358ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.558+0000 7fe358ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:38.539 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:38.539 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:38.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 10 librados: watch_flush enter 2024-03-29T11:01:38.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 10 librados: watch_flush exit 2024-03-29T11:01:38.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:01:38.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:01:38.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe34806dbe0 legacy=0x7fe3480700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 10 monclient: shutdown 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 >> v1:172.21.15.45:6790/0 conn(0x7fe35c13ef30 legacy=0x7fe35c06ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 shutdown_connections 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.562+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 >> 172.21.15.45:0/2942472575 conn(0x7fe35c06b660 msgr2=0x7fe35c14fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.563+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 shutdown_connections 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.563+0000 7fe3627af640 1 -- 172.21.15.45:0/2942472575 wait complete. 2024-03-29T11:01:38.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:38.564+0000 7fe3627af640 1 librados: shutdown 2024-03-29T11:01:38.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:45.367 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:45.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.522+0000 7f081a84e640 10 monclient: get_monmap_and_config 2024-03-29T11:01:45.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.522+0000 7f081a84e640 10 monclient: build_initial_monmap 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.522+0000 7f081a84e640 10 monclient: monmap: 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:45.523673+0000 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:45.523673+0000 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:45.503 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.523+0000 7f081a84e640 1 Processor -- start 2024-03-29T11:01:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.523+0000 7f081a84e640 1 -- start start 2024-03-29T11:01:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.523+0000 7f081a84e640 10 monclient: init 2024-03-29T11:01:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient(hunting): picked mon.noname-c con 0x7f08141457f0 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient(hunting): picked mon.noname-b con 0x7f081413f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient(hunting): picked mon.noname-a con 0x7f08140a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140aaef0 con 0x7f08140a9d70 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814145bd0 con 0x7f08141457f0 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814145eb0 con 0x7f081413f650 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.524+0000 7f081a84e640 10 monclient(hunting): authenticate will time out at 1379.793879s 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f08137fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f081413f650 0x7f08140b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57746/0 (socket says 172.21.15.45:57746) 2024-03-29T11:01:45.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f08137fe640 1 -- 172.21.15.45:0/2701052450 learned_addr learned my addr 172.21.15.45:0/2701052450 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176054181 0 0) 0x7f0814145eb0 con 0x7f081413f650 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f0812ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f0812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f0812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f0812ffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.525+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003520 con 0x7f081413f650 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.526+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604912704 0 0) 0x7f07fc003520 con 0x7f081413f650 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.526+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 >> v1:172.21.15.116:6789/0 conn(0x7f08141457f0 legacy=0x7f081413ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:01:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.526+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 >> v1:172.21.15.45:6789/0 conn(0x7f08140a9d70 legacy=0x7f081414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 1 monclient: found mon.noname-b 2024-03-29T11:01:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:01:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814146340 con 0x7f081413f650 2024-03-29T11:01:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0808001690 con 0x7f081413f650 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: dump: 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:45.509 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f081a84e640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f081a84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0808001860 con 0x7f081413f650 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 1 -- 172.21.15.45:0/2701052450 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0808001cb0 con 0x7f081413f650 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.528+0000 7f0812ffd640 10 monclient: dump: 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.510 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:45.511 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:45.511 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:45.511 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:45.511 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:45.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.529+0000 7f081a84e640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:45.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.529+0000 7f081a84e640 4 set_mon_vals no callback set 2024-03-29T11:01:45.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.529+0000 7f081a84e640 10 monclient: shutdown 2024-03-29T11:01:45.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.529+0000 7f081a84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.529+0000 7f081a84e640 1 -- 172.21.15.45:0/2701052450 >> v1:172.21.15.45:6790/0 conn(0x7f081413f650 legacy=0x7f08140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.530+0000 7f081a84e640 1 -- 172.21.15.45:0/2701052450 shutdown_connections 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.530+0000 7f081a84e640 1 -- 172.21.15.45:0/2701052450 >> 172.21.15.45:0/2701052450 conn(0x7f0814066a00 msgr2=0x7f0814066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.530+0000 7f081a84e640 1 -- 172.21.15.45:0/2701052450 shutdown_connections 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.530+0000 7f081a84e640 1 -- 172.21.15.45:0/2701052450 wait complete. 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.531+0000 7f081a84e640 10 monclient: build_initial_monmap 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.531+0000 7f081a84e640 10 monclient: monmap: 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:45.515 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.531+0000 7f081a84e640 1 librados: starting msgr at 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.531+0000 7f081a84e640 1 librados: starting objecter 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 1 Processor -- start 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 1 -- start start 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 1 librados: setting wanted keys 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 1 librados: calling monclient init 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 10 monclient: init 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.532+0000 7f081a84e640 10 monclient(hunting): picked mon.noname-b con 0x7f08140a9d70 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 10 monclient(hunting): picked mon.noname-a con 0x7f08141457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:45.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 10 monclient(hunting): picked mon.noname-c con 0x7f08141448c0 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814141f60 con 0x7f08141457f0 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141418b0 con 0x7f08140a9d70 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814141a40 con 0x7f08141448c0 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f081a84e640 10 monclient(hunting): authenticate will time out at 1379.802449s 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f0813fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f08140a9d70 0x7f081413fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:59458/0 (socket says 172.21.15.45:59458) 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.533+0000 7f0813fff640 1 -- 172.21.15.45:0/4085765461 learned_addr learned my addr 172.21.15.45:0/4085765461 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 client.?.objecter ms_handle_connect 0x7f08141448c0 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 client.?.objecter ms_handle_connect 0x7f08140a9d70 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576725809 0 0) 0x7f0814141a40 con 0x7f08141448c0 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800004eb0 con 0x7f08141448c0 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549215062 0 0) 0x7f08141418b0 con 0x7f08140a9d70 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0814141a40 con 0x7f08140a9d70 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989367192 0 0) 0x7f0800004eb0 con 0x7f08141448c0 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08141418b0 con 0x7f08141448c0 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f08040017f0 con 0x7f08141448c0 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:45.518 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.534+0000 7f0810ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.535+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2949061415 0 0) 0x7f08141418b0 con 0x7f08141448c0 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.535+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 >> v1:172.21.15.116:6789/0 conn(0x7f08140a9d70 legacy=0x7f081413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.535+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 >> v1:172.21.15.45:6789/0 conn(0x7f08141457f0 legacy=0x7f08141404b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 1 monclient: found mon.c 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814141bd0 con 0x7f08141448c0 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 10 monclient: _finish_auth 0 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f0810ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f081a84e640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f081a84e640 10 monclient: _renew_subs 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f081a84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.536+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08141428f0 con 0x7f08141448c0 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f081a84e640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:45.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f081a84e640 10 monclient: _renew_subs 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f081a84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141e32e0 con 0x7f08141448c0 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f081a84e640 1 librados: init done 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0804001d00 con 0x7f08141448c0 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f0810ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f0804003790 con 0x7f08141448c0 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f0810ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f0810ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f0810ff9640 10 monclient: dump: 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.524 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.537+0000 7f08127fc640 4 set_mon_vals no callback set 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.538+0000 7f081a84e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.538+0000 7f081a84e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.538+0000 7f081a84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.538+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f4005080 con 0x7f08141448c0 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.539+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f0804019760 con 0x7f08141448c0 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.540+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f0804083670 con 0x7f08141448c0 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.540+0000 7f0810ff9640 10 client.4607.objecter ms_dispatch 0x7f0814000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.540+0000 7f0810ff9640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.540+0000 7f0810ff9640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.540+0000 7f0810ff9640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:01:45.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.540+0000 7f0813fff640 10 monclient: get_auth_request con 0x7f080006dad0 auth_method 0 2024-03-29T11:01:45.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.541+0000 7f0810ff9640 10 client.4607.objecter ms_handle_connect 0x7f080006dad0 2024-03-29T11:01:45.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.546+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f080404efb0 con 0x7f08141448c0 2024-03-29T11:01:45.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.546+0000 7f0810ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.546+0000 7f0810ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:45.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.860+0000 7f081a84e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.860+0000 7f081a84e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.860+0000 7f081a84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:01:45.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.860+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07f4005330 con 0x7f08141448c0 2024-03-29T11:01:45.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.861+0000 7f0810ff9640 1 -- 172.21.15.45:0/4085765461 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f0804050ee0 con 0x7f08141448c0 2024-03-29T11:01:45.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.861+0000 7f0810ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.861+0000 7f0810ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:45.841 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:45.841 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:45.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.864+0000 7f081a84e640 10 librados: watch_flush enter 2024-03-29T11:01:45.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.864+0000 7f081a84e640 10 librados: watch_flush exit 2024-03-29T11:01:45.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.864+0000 7f081a84e640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:01:45.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.864+0000 7f081a84e640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:01:45.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 >> v1:172.21.15.116:6816/3290579615 conn(0x7f080006dad0 legacy=0x7f080006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 10 monclient: shutdown 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 >> v1:172.21.15.45:6790/0 conn(0x7f08141448c0 legacy=0x7f08141e1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 shutdown_connections 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 >> 172.21.15.45:0/4085765461 conn(0x7f0814066a00 msgr2=0x7f08140b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 shutdown_connections 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 -- 172.21.15.45:0/4085765461 wait complete. 2024-03-29T11:01:45.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:45.865+0000 7f081a84e640 1 librados: shutdown 2024-03-29T11:01:45.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:52.666 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:52.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.830+0000 7f4863134640 10 monclient: get_monmap_and_config 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.830+0000 7f4863134640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.830+0000 7f4863134640 10 monclient: monmap: 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:01:52.832201+0000 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:01:52.832201+0000 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.831+0000 7f4863134640 1 Processor -- start 2024-03-29T11:01:52.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.831+0000 7f4863134640 1 -- start start 2024-03-29T11:01:52.812 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.831+0000 7f4863134640 10 monclient: init 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient(hunting): picked mon.noname-c con 0x7f485c0b2790 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient(hunting): picked mon.noname-a con 0x7f485c143c80 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient(hunting): picked mon.noname-b con 0x7f485c146e30 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c146290 con 0x7f485c143c80 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0b3970 con 0x7f485c0b2790 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c06ee80 con 0x7f485c146e30 2024-03-29T11:01:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4863134640 10 monclient(hunting): authenticate will time out at 1387.102696s 2024-03-29T11:01:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4862132640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f485c0b2790 0x7f485c143570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:59622/0 (socket says 172.21.15.45:59622) 2024-03-29T11:01:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.832+0000 7f4862132640 1 -- 172.21.15.45:0/1485065226 learned_addr learned my addr 172.21.15.45:0/1485065226 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264422033 0 0) 0x7f485c06ee80 con 0x7f485c146e30 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003520 con 0x7f485c146e30 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991488207 0 0) 0x7f485c146290 con 0x7f485c143c80 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c06ee80 con 0x7f485c143c80 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700641007 0 0) 0x7f485c0b3970 con 0x7f485c0b2790 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.833+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c146290 con 0x7f485c0b2790 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1721074920 0 0) 0x7f4848003520 con 0x7f485c146e30 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 >> v1:172.21.15.116:6789/0 conn(0x7f485c0b2790 legacy=0x7f485c143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 >> v1:172.21.15.45:6789/0 conn(0x7f485c143c80 legacy=0x7f485c146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 monclient: found mon.noname-b 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c06f160 con 0x7f485c146e30 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f484c001690 con 0x7f485c146e30 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4863134640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4863134640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: dump: 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4863134640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f484c0018e0 con 0x7f485c146e30 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 1 -- 172.21.15.45:0/1485065226 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f484c001c70 con 0x7f485c146e30 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4861130640 10 monclient: dump: 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4863134640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.834+0000 7f4863134640 4 set_mon_vals no callback set 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.835+0000 7f4863134640 10 monclient: shutdown 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.835+0000 7f4863134640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.835+0000 7f4863134640 1 -- 172.21.15.45:0/1485065226 >> v1:172.21.15.45:6790/0 conn(0x7f485c146e30 legacy=0x7f485c151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.835+0000 7f4863134640 1 -- 172.21.15.45:0/1485065226 shutdown_connections 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.835+0000 7f4863134640 1 -- 172.21.15.45:0/1485065226 >> 172.21.15.45:0/1485065226 conn(0x7f485c06d070 msgr2=0x7f485c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.835+0000 7f4863134640 1 -- 172.21.15.45:0/1485065226 shutdown_connections 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.836+0000 7f4863134640 1 -- 172.21.15.45:0/1485065226 wait complete. 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 10 monclient: monmap: 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 1 librados: starting msgr at 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 1 librados: starting objecter 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 1 Processor -- start 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 1 -- start start 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 1 librados: setting wanted keys 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 1 librados: calling monclient init 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.837+0000 7f4863134640 10 monclient: init 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.838+0000 7f4863134640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0b2790 addr v1:172.21.15.116:6789/0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 10 monclient(hunting): picked mon.noname-a con 0x7f485c143c80 addr v1:172.21.15.45:6789/0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 10 monclient(hunting): picked mon.noname-c con 0x7f485c146e30 addr v1:172.21.15.45:6790/0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c1db3b0 con 0x7f485c143c80 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c1db690 con 0x7f485c0b2790 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c1dc230 con 0x7f485c146e30 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4863134640 10 monclient(hunting): authenticate will time out at 1387.109112s 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4862132640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f485c0b2790 0x7f485c148cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:59880/0 (socket says 172.21.15.45:59880) 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4862132640 1 -- 172.21.15.45:0/2803655587 learned_addr learned my addr 172.21.15.45:0/2803655587 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 client.?.objecter ms_handle_connect 0x7f485c143c80 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 client.?.objecter ms_handle_connect 0x7f485c146e30 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964197003 0 0) 0x7f485c1db3b0 con 0x7f485c143c80 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840004eb0 con 0x7f485c143c80 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104115629 0 0) 0x7f485c1dc230 con 0x7f485c146e30 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48400051d0 con 0x7f485c146e30 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.839+0000 7f4852ffd640 10 client.?.objecter ms_handle_connect 0x7f485c0b2790 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34816929 0 0) 0x7f485c1db690 con 0x7f485c0b2790 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c1dc230 con 0x7f485c0b2790 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2076422636 0 0) 0x7f4840004eb0 con 0x7f485c143c80 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f485c1db690 con 0x7f485c143c80 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4858001930 con 0x7f485c143c80 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): dump: 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954312794 0 0) 0x7f485c1dc230 con 0x7f485c0b2790 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4840004eb0 con 0x7f485c0b2790 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4854001c30 con 0x7f485c0b2790 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient(hunting): dump: 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3105317535 0 0) 0x7f485c1db690 con 0x7f485c143c80 2024-03-29T11:01:52.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 >> v1:172.21.15.45:6790/0 conn(0x7f485c146e30 legacy=0x7f485c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 >> v1:172.21.15.116:6789/0 conn(0x7f485c0b2790 legacy=0x7f485c148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 monclient: found mon.a 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c14a300 con 0x7f485c143c80 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 10 monclient: _renew_subs 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c13c420 con 0x7f485c143c80 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 10 monclient: _renew_subs 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.840+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c1480f0 con 0x7f485c143c80 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4863134640 1 librados: init done 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4858003610 con 0x7f485c143c80 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4852ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4858003920 con 0x7f485c143c80 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4852ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f4852ffd640 10 monclient: dump: 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:01:52.833 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.841+0000 7f486092f640 4 set_mon_vals no callback set 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.842+0000 7f4863134640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.842+0000 7f4863134640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.842+0000 7f4863134640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.842+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4844005080 con 0x7f485c143c80 2024-03-29T11:01:52.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.844+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f48580198f0 con 0x7f485c143c80 2024-03-29T11:01:52.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.845+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f4858083730 con 0x7f485c143c80 2024-03-29T11:01:52.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.845+0000 7f4852ffd640 10 client.4548.objecter ms_dispatch 0x7f485c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:52.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.845+0000 7f4852ffd640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.845+0000 7f4852ffd640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.845+0000 7f4852ffd640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.845+0000 7f4862132640 10 monclient: get_auth_request con 0x7f484006e120 auth_method 0 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.846+0000 7f4852ffd640 10 client.4548.objecter ms_handle_connect 0x7f484006e120 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.853+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f485804ee40 con 0x7f485c143c80 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.853+0000 7f4852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:52.853+0000 7f4852ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.231+0000 7f4863134640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.231+0000 7f4863134640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.231+0000 7f4863134640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.231+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4844005330 con 0x7f485c143c80 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.232+0000 7f4852ffd640 1 -- 172.21.15.45:0/2803655587 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f4858050f40 con 0x7f485c143c80 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.232+0000 7f4852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.232+0000 7f4852ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:53.213 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:01:53.213 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:53.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.235+0000 7f4863134640 10 librados: watch_flush enter 2024-03-29T11:01:53.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.235+0000 7f4863134640 10 librados: watch_flush exit 2024-03-29T11:01:53.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.235+0000 7f4863134640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T11:01:53.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.235+0000 7f4863134640 10 client.4548.objecter successfully canceled tick 2024-03-29T11:01:53.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.236+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 >> v1:172.21.15.116:6816/3290579615 conn(0x7f484006e120 legacy=0x7f48400705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:53.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.236+0000 7f4863134640 10 monclient: shutdown 2024-03-29T11:01:53.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.236+0000 7f4863134640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:53.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.236+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 >> v1:172.21.15.45:6789/0 conn(0x7f485c143c80 legacy=0x7f485c1dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:53.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.236+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 shutdown_connections 2024-03-29T11:01:53.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.236+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 >> 172.21.15.45:0/2803655587 conn(0x7f485c06d070 msgr2=0x7f485c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:53.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.237+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 shutdown_connections 2024-03-29T11:01:53.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.237+0000 7f4863134640 1 -- 172.21.15.45:0/2803655587 wait complete. 2024-03-29T11:01:53.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:01:53.237+0000 7f4863134640 1 librados: shutdown 2024-03-29T11:01:53.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:00.039 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.210+0000 7f4b3997a640 10 monclient: get_monmap_and_config 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.210+0000 7f4b3997a640 10 monclient: build_initial_monmap 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.210+0000 7f4b3997a640 10 monclient: monmap: 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:00.212303+0000 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:00.212303+0000 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.216+0000 7f4b3997a640 1 Processor -- start 2024-03-29T11:02:00.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.216+0000 7f4b3997a640 1 -- start start 2024-03-29T11:02:00.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.216+0000 7f4b3997a640 10 monclient: init 2024-03-29T11:02:00.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3406d9d0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient(hunting): picked mon.noname-a con 0x7f4b3413ef30 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient(hunting): picked mon.noname-b con 0x7f4b340b07d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340aed10 con 0x7f4b3413ef30 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340b0c40 con 0x7f4b3406d9d0 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340ab1b0 con 0x7f4b340b07d0 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:00.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.217+0000 7f4b3997a640 10 monclient(hunting): authenticate will time out at 1394.487259s 2024-03-29T11:02:00.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.220+0000 7f4b327fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4b3413ef30 0x7f4b340af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:57731/0 (socket says 172.21.15.45:57731) 2024-03-29T11:02:00.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.220+0000 7f4b327fc640 1 -- 172.21.15.45:0/2065804308 learned_addr learned my addr 172.21.15.45:0/2065804308 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156164738 0 0) 0x7f4b340aed10 con 0x7f4b3413ef30 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b1c003520 con 0x7f4b3413ef30 2024-03-29T11:02:00.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877106149 0 0) 0x7f4b340ab1b0 con 0x7f4b340b07d0 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.223+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b340aed10 con 0x7f4b340b07d0 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310881458 0 0) 0x7f4b340aed10 con 0x7f4b340b07d0 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 >> v1:172.21.15.116:6789/0 conn(0x7f4b3406d9d0 legacy=0x7f4b3413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:00.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 >> v1:172.21.15.45:6789/0 conn(0x7f4b3413ef30 legacy=0x7f4b340af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 monclient: found mon.noname-b 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b340ab490 con 0x7f4b340b07d0 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4b28001520 con 0x7f4b340b07d0 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: dump: 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:00.206 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b3997a640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b3997a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b28001700 con 0x7f4b340b07d0 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 1 -- 172.21.15.45:0/2065804308 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4b28001b50 con 0x7f4b340b07d0 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b31ffb640 10 monclient: dump: 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b3997a640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:00.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.224+0000 7f4b3997a640 4 set_mon_vals no callback set 2024-03-29T11:02:00.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.225+0000 7f4b3997a640 10 monclient: shutdown 2024-03-29T11:02:00.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.225+0000 7f4b3997a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:00.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.225+0000 7f4b3997a640 1 -- 172.21.15.45:0/2065804308 >> v1:172.21.15.45:6790/0 conn(0x7f4b340b07d0 legacy=0x7f4b3414bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.229+0000 7f4b3997a640 1 -- 172.21.15.45:0/2065804308 shutdown_connections 2024-03-29T11:02:00.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.229+0000 7f4b3997a640 1 -- 172.21.15.45:0/2065804308 >> 172.21.15.45:0/2065804308 conn(0x7f4b3406b660 msgr2=0x7f4b3406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.231+0000 7f4b3997a640 1 -- 172.21.15.45:0/2065804308 shutdown_connections 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.232+0000 7f4b3997a640 1 -- 172.21.15.45:0/2065804308 wait complete. 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.233+0000 7f4b3997a640 10 monclient: build_initial_monmap 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.233+0000 7f4b3997a640 10 monclient: monmap: 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.215 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:00.216 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:00.216 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:00.216 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.233+0000 7f4b3997a640 1 librados: starting msgr at 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.233+0000 7f4b3997a640 1 librados: starting objecter 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.233+0000 7f4b3997a640 1 Processor -- start 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.234+0000 7f4b3997a640 1 -- start start 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.234+0000 7f4b3997a640 1 librados: setting wanted keys 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.234+0000 7f4b3997a640 1 librados: calling monclient init 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.234+0000 7f4b3997a640 10 monclient: init 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.234+0000 7f4b3997a640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.234+0000 7f4b3997a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 10 monclient(hunting): picked mon.noname-b con 0x7f4b3406d9d0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 10 monclient(hunting): picked mon.noname-c con 0x7f4b340b07d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:00.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 10 monclient(hunting): picked mon.noname-a con 0x7f4b3413ef30 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:00.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b341385f0 con 0x7f4b3413ef30 2024-03-29T11:02:00.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b341377e0 con 0x7f4b3406d9d0 2024-03-29T11:02:00.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b34137ac0 con 0x7f4b340b07d0 2024-03-29T11:02:00.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:00.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.235+0000 7f4b3997a640 10 monclient(hunting): authenticate will time out at 1394.505049s 2024-03-29T11:02:00.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.238+0000 7f4b337fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f4b3413ef30 0x7f4b34147660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58201/0 (socket says 172.21.15.45:58201) 2024-03-29T11:02:00.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.238+0000 7f4b337fe640 1 -- 172.21.15.45:0/3159832555 learned_addr learned my addr 172.21.15.45:0/3159832555 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:00.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.238+0000 7f4b1bfff640 10 client.?.objecter ms_handle_connect 0x7f4b3413ef30 2024-03-29T11:02:00.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 client.?.objecter ms_handle_connect 0x7f4b340b07d0 2024-03-29T11:02:00.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 client.?.objecter ms_handle_connect 0x7f4b3406d9d0 2024-03-29T11:02:00.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055562990 0 0) 0x7f4b341385f0 con 0x7f4b3413ef30 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b20004eb0 con 0x7f4b3413ef30 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693475145 0 0) 0x7f4b34137ac0 con 0x7f4b340b07d0 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.239+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b341385f0 con 0x7f4b340b07d0 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.240+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045963079 0 0) 0x7f4b341377e0 con 0x7f4b3406d9d0 2024-03-29T11:02:00.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.240+0000 7f4b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.240+0000 7f4b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.240+0000 7f4b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.240+0000 7f4b1bfff640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b34137ac0 con 0x7f4b3406d9d0 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2620287063 0 0) 0x7f4b341385f0 con 0x7f4b340b07d0 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b341377e0 con 0x7f4b340b07d0 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1316464154 0 0) 0x7f4b20004eb0 con 0x7f4b3413ef30 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b341385f0 con 0x7f4b3413ef30 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4b2c001c50 con 0x7f4b340b07d0 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): dump: 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:00.227 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4b28001a50 con 0x7f4b3413ef30 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 10 monclient(hunting): dump: 2024-03-29T11:02:00.228 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.241+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2863021268 0 0) 0x7f4b341385f0 con 0x7f4b3413ef30 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 >> v1:172.21.15.45:6790/0 conn(0x7f4b340b07d0 legacy=0x7f4b34135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 >> v1:172.21.15.116:6789/0 conn(0x7f4b3406d9d0 legacy=0x7f4b34137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 1 monclient: found mon.a 2024-03-29T11:02:00.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3413b270 con 0x7f4b3413ef30 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.242+0000 7f4b3997a640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 10 monclient: _renew_subs 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b34135d20 con 0x7f4b3413ef30 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 10 monclient: _renew_subs 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3413ae90 con 0x7f4b3413ef30 2024-03-29T11:02:00.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.243+0000 7f4b3997a640 1 librados: init done 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.244+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b28001060 con 0x7f4b3413ef30 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.244+0000 7f4b1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.244+0000 7f4b3997a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.244+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f4b28003a40 con 0x7f4b3413ef30 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.245+0000 7f4b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.245+0000 7f4b1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.245+0000 7f4b1bfff640 10 monclient: dump: 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:00.231 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:00.232 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:00.232 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:00.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.244+0000 7f4b317fa640 4 set_mon_vals no callback set 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.245+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f4b28003c60 con 0x7f4b3413ef30 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b3997a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b3997a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b34075f10 con 0x7f4b3413ef30 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3474427043 0 0) 0x7f4b28084a40 con 0x7f4b3413ef30 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b1bfff640 10 client.4554.objecter ms_dispatch 0x7f4b34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:00.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b1bfff640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:00.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b1bfff640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:00.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.246+0000 7f4b1bfff640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:02:00.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.248+0000 7f4b32ffd640 10 monclient: get_auth_request con 0x7f4b2006e050 auth_method 0 2024-03-29T11:02:00.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.250+0000 7f4b1bfff640 10 client.4554.objecter ms_handle_connect 0x7f4b2006e050 2024-03-29T11:02:00.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.260+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b28050320 con 0x7f4b3413ef30 2024-03-29T11:02:00.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.260+0000 7f4b1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.260+0000 7f4b1bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:00.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.599+0000 7f4b3997a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.599+0000 7f4b3997a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.600+0000 7f4b3997a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:00.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.600+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b34072330 con 0x7f4b3413ef30 2024-03-29T11:02:00.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.601+0000 7f4b1bfff640 1 -- 172.21.15.45:0/3159832555 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1635500216) 0x7f4b28052250 con 0x7f4b3413ef30 2024-03-29T11:02:00.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.601+0000 7f4b1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.601+0000 7f4b1bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:00.582 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:00.582 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T10:52:03.675545+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:52:03.675545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.603+0000 7f4b3997a640 10 librados: watch_flush enter 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.603+0000 7f4b3997a640 10 librados: watch_flush exit 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.603+0000 7f4b3997a640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.603+0000 7f4b3997a640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.603+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 >> v1:172.21.15.116:6816/3290579615 conn(0x7f4b2006e050 legacy=0x7f4b20070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.603+0000 7f4b3997a640 10 monclient: shutdown 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 >> v1:172.21.15.45:6789/0 conn(0x7f4b3413ef30 legacy=0x7f4b34147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 shutdown_connections 2024-03-29T11:02:00.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 >> 172.21.15.45:0/3159832555 conn(0x7f4b3406b660 msgr2=0x7f4b3413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:00.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 shutdown_connections 2024-03-29T11:02:00.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 1 -- 172.21.15.45:0/3159832555 wait complete. 2024-03-29T11:02:00.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:00.604+0000 7f4b3997a640 1 librados: shutdown 2024-03-29T11:02:00.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:07.407 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.576+0000 7fe6a72d0640 10 monclient: get_monmap_and_config 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.576+0000 7fe6a72d0640 10 monclient: build_initial_monmap 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.576+0000 7fe6a72d0640 10 monclient: monmap: 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:07.577204+0000 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:07.577204+0000 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:07.556 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:07.557 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:07.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.578+0000 7fe6a72d0640 1 Processor -- start 2024-03-29T11:02:07.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.578+0000 7fe6a72d0640 1 -- start start 2024-03-29T11:02:07.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.578+0000 7fe6a72d0640 10 monclient: init 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.579+0000 7fe6a72d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.579+0000 7fe6a72d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.579+0000 7fe6a72d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe698038c60 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe6980324b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe6980d11e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6980caf80 con 0x7fe6980d11e0 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe698039850 con 0x7fe698038c60 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6980c78d0 con 0x7fe6980324b0 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:07.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.580+0000 7fe6a72d0640 10 monclient(hunting): authenticate will time out at 1401.849179s 2024-03-29T11:02:07.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.583+0000 7fe6a5acd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6980324b0 0x7fe6980cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58433/0 (socket says 172.21.15.45:58433) 2024-03-29T11:02:07.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.583+0000 7fe6a5acd640 1 -- 172.21.15.45:0/2422218201 learned_addr learned my addr 172.21.15.45:0/2422218201 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:07.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.586+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331870191 0 0) 0x7fe6980caf80 con 0x7fe6980d11e0 2024-03-29T11:02:07.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.586+0000 7fe6a52cc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.586+0000 7fe6a52cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.586+0000 7fe6a52cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.586+0000 7fe6a52cc640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:02:07.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.586+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe68c003520 con 0x7fe6980d11e0 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608394953 0 0) 0x7fe6980c78d0 con 0x7fe6980324b0 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6980caf80 con 0x7fe6980324b0 2024-03-29T11:02:07.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926498376 0 0) 0x7fe698039850 con 0x7fe698038c60 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6980c78d0 con 0x7fe698038c60 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168483373 0 0) 0x7fe6980caf80 con 0x7fe6980324b0 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 >> v1:172.21.15.116:6789/0 conn(0x7fe698038c60 legacy=0x7fe6980c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 >> v1:172.21.15.45:6789/0 conn(0x7fe6980d11e0 legacy=0x7fe6980d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 monclient: found mon.noname-b 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6980c7a60 con 0x7fe6980324b0 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient: _finish_auth 0 2024-03-29T11:02:07.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.587+0000 7fe6a52cc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:07.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:07.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe69c001670 con 0x7fe6980324b0 2024-03-29T11:02:07.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a72d0640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:02:07.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a72d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 10 monclient: dump: 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:07.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a72d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe69c0018c0 con 0x7fe6980324b0 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a52cc640 1 -- 172.21.15.45:0/2422218201 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe69c001c50 con 0x7fe6980324b0 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a72d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.588+0000 7fe6a72d0640 4 set_mon_vals no callback set 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 10 monclient: shutdown 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2422218201 >> v1:172.21.15.45:6790/0 conn(0x7fe6980324b0 legacy=0x7fe6980cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a52cc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2422218201 shutdown_connections 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2422218201 >> 172.21.15.45:0/2422218201 conn(0x7fe698005c20 msgr2=0x7fe698006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2422218201 shutdown_connections 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.589+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2422218201 wait complete. 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.590+0000 7fe6a72d0640 10 monclient: build_initial_monmap 2024-03-29T11:02:07.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.590+0000 7fe6a72d0640 10 monclient: monmap: 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:07.575 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:07.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.591+0000 7fe6a72d0640 1 librados: starting msgr at 2024-03-29T11:02:07.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.594+0000 7fe6a72d0640 1 librados: starting objecter 2024-03-29T11:02:07.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.594+0000 7fe6a72d0640 1 Processor -- start 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 1 -- start start 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 1 librados: setting wanted keys 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 1 librados: calling monclient init 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 10 monclient: init 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.595+0000 7fe6a72d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe6980324b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe698038c60 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe6980d11e0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69815fee0 con 0x7fe698038c60 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe698036cb0 con 0x7fe6980d11e0 2024-03-29T11:02:07.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6980c8c50 con 0x7fe6980324b0 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.596+0000 7fe6a72d0640 10 monclient(hunting): authenticate will time out at 1401.865040s 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe6a5acd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe698038c60 0x7fe6980be280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60477/0 (socket says 172.21.15.45:60477) 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe6a5acd640 1 -- 172.21.15.45:0/2930747961 learned_addr learned my addr 172.21.15.45:0/2930747961 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 10 client.?.objecter ms_handle_connect 0x7fe698038c60 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810958563 0 0) 0x7fe69815fee0 con 0x7fe698038c60 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:02:07.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.599+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe684004eb0 con 0x7fe698038c60 2024-03-29T11:02:07.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 10 client.?.objecter ms_handle_connect 0x7fe6980324b0 2024-03-29T11:02:07.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:07.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774510996 0 0) 0x7fe6980c8c50 con 0x7fe6980324b0 2024-03-29T11:02:07.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:02:07.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.601+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe69815fee0 con 0x7fe6980324b0 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3859628261 0 0) 0x7fe684004eb0 con 0x7fe698038c60 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6980c8c50 con 0x7fe698038c60 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 711789610 0 0) 0x7fe69815fee0 con 0x7fe6980324b0 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe684004eb0 con 0x7fe6980324b0 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe69c0016f0 con 0x7fe698038c60 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:07.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): dump: 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe6a004fdd0 con 0x7fe6980324b0 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient(hunting): dump: 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3583143462 0 0) 0x7fe6980c8c50 con 0x7fe698038c60 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 >> v1:172.21.15.45:6790/0 conn(0x7fe6980324b0 legacy=0x7fe698160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 >> v1:172.21.15.116:6789/0 conn(0x7fe6980d11e0 legacy=0x7fe698168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 monclient: found mon.a 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698036f20 con 0x7fe698038c60 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 10 monclient: _renew_subs 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe698034d30 con 0x7fe698038c60 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:07.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 10 monclient: _renew_subs 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe698034a30 con 0x7fe698038c60 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a72d0640 1 librados: init done 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe69c0011f0 con 0x7fe698038c60 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe69c004db0 con 0x7fe698038c60 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe696ffd640 10 monclient: dump: 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.595 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.603+0000 7fe6a4acb640 4 set_mon_vals no callback set 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.605+0000 7fe6a72d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.605+0000 7fe6a72d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.605+0000 7fe6a72d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.605+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe688005080 con 0x7fe698038c60 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.606+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe69c004f10 con 0x7fe698038c60 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.607+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7fe69c084d20 con 0x7fe698038c60 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.607+0000 7fe696ffd640 10 client.4560.objecter ms_dispatch 0x7fe6a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.607+0000 7fe696ffd640 3 client.4560.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.607+0000 7fe696ffd640 3 client.4560.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:07.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.607+0000 7fe696ffd640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:02:07.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.608+0000 7fe6a62ce640 10 monclient: get_auth_request con 0x7fe68406dc00 auth_method 0 2024-03-29T11:02:07.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.609+0000 7fe696ffd640 10 client.4560.objecter ms_handle_connect 0x7fe68406dc00 2024-03-29T11:02:07.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.620+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe69c050600 con 0x7fe698038c60 2024-03-29T11:02:07.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.620+0000 7fe696ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.620+0000 7fe696ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:07.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.944+0000 7fe6a72d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.944+0000 7fe6a72d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.944+0000 7fe6a72d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:07.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.944+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe688005330 con 0x7fe698038c60 2024-03-29T11:02:07.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.945+0000 7fe696ffd640 1 -- 172.21.15.45:0/2930747961 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7fe69c052530 con 0x7fe698038c60 2024-03-29T11:02:07.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.945+0000 7fe696ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.945+0000 7fe696ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:07.926 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:07.926 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.949+0000 7fe6a72d0640 10 librados: watch_flush enter 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.949+0000 7fe6a72d0640 10 librados: watch_flush exit 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe68406dc00 legacy=0x7fe6840700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 10 monclient: shutdown 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.929 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 >> v1:172.21.15.45:6789/0 conn(0x7fe698038c60 legacy=0x7fe6980be280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 shutdown_connections 2024-03-29T11:02:07.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 >> 172.21.15.45:0/2930747961 conn(0x7fe698005c20 msgr2=0x7fe6980043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 shutdown_connections 2024-03-29T11:02:07.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 -- 172.21.15.45:0/2930747961 wait complete. 2024-03-29T11:02:07.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:07.950+0000 7fe6a72d0640 1 librados: shutdown 2024-03-29T11:02:07.951 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:14.758 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:14.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.922+0000 7fe449425640 10 monclient: get_monmap_and_config 2024-03-29T11:02:14.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.922+0000 7fe449425640 10 monclient: build_initial_monmap 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.922+0000 7fe449425640 10 monclient: monmap: 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:14.922920+0000 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:14.922920+0000 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:14.903 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:14.904 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.923+0000 7fe449425640 1 Processor -- start 2024-03-29T11:02:14.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 1 -- start start 2024-03-29T11:02:14.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient: init 2024-03-29T11:02:14.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:14.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:14.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient(hunting): picked mon.noname-a con 0x7fe4440ae250 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:14.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient(hunting): picked mon.noname-c con 0x7fe4440a9720 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:14.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient(hunting): picked mon.noname-b con 0x7fe44414bba0 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:14.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440a9b80 con 0x7fe4440ae250 2024-03-29T11:02:14.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440aa7b0 con 0x7fe4440a9720 2024-03-29T11:02:14.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440afbf0 con 0x7fe44414bba0 2024-03-29T11:02:14.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:14.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.927+0000 7fe449425640 10 monclient(hunting): authenticate will time out at 1409.196923s 2024-03-29T11:02:14.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.930+0000 7fe4427fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe4440a9720 0x7fe44413f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:51571/0 (socket says 172.21.15.45:51571) 2024-03-29T11:02:14.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.930+0000 7fe4427fc640 1 -- 172.21.15.45:0/1272071761 learned_addr learned my addr 172.21.15.45:0/1272071761 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:14.915 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235703027 0 0) 0x7fe4440a9b80 con 0x7fe4440ae250 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c003520 con 0x7fe4440ae250 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043369420 0 0) 0x7fe4440aa7b0 con 0x7fe4440a9720 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4440a9b80 con 0x7fe4440a9720 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 842014634 0 0) 0x7fe42c003520 con 0x7fe4440ae250 2024-03-29T11:02:14.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.934+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 >> v1:172.21.15.45:6790/0 conn(0x7fe44414bba0 legacy=0x7fe44414df90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 >> v1:172.21.15.116:6789/0 conn(0x7fe4440a9720 legacy=0x7fe44413f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 1 monclient: found mon.noname-a 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe444143280 con 0x7fe4440ae250 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe43c001a10 con 0x7fe4440ae250 2024-03-29T11:02:14.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.936+0000 7fe441ffb640 10 monclient: dump: 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe449425640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:02:14.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe449425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe43c001b70 con 0x7fe4440ae250 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe441ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe441ffb640 1 -- 172.21.15.45:0/1272071761 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe43c001f40 con 0x7fe4440ae250 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe441ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe441ffb640 10 monclient: dump: 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe449425640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:14.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.937+0000 7fe449425640 4 set_mon_vals no callback set 2024-03-29T11:02:14.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.938+0000 7fe449425640 10 monclient: shutdown 2024-03-29T11:02:14.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.939+0000 7fe449425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:14.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.939+0000 7fe449425640 1 -- 172.21.15.45:0/1272071761 >> v1:172.21.15.45:6789/0 conn(0x7fe4440ae250 legacy=0x7fe444142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.939+0000 7fe449425640 1 -- 172.21.15.45:0/1272071761 shutdown_connections 2024-03-29T11:02:14.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.939+0000 7fe449425640 1 -- 172.21.15.45:0/1272071761 >> 172.21.15.45:0/1272071761 conn(0x7fe4440b35e0 msgr2=0x7fe4440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:14.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.940+0000 7fe449425640 1 -- 172.21.15.45:0/1272071761 shutdown_connections 2024-03-29T11:02:14.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.942+0000 7fe449425640 1 -- 172.21.15.45:0/1272071761 wait complete. 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.944+0000 7fe449425640 10 monclient: build_initial_monmap 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.944+0000 7fe449425640 10 monclient: monmap: 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:14.924 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.944+0000 7fe449425640 1 librados: starting msgr at 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.944+0000 7fe449425640 1 librados: starting objecter 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.944+0000 7fe449425640 1 Processor -- start 2024-03-29T11:02:14.925 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.945+0000 7fe449425640 1 -- start start 2024-03-29T11:02:14.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.945+0000 7fe449425640 1 librados: setting wanted keys 2024-03-29T11:02:14.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.945+0000 7fe449425640 1 librados: calling monclient init 2024-03-29T11:02:14.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.945+0000 7fe449425640 10 monclient: init 2024-03-29T11:02:14.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.946+0000 7fe449425640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:14.926 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.946+0000 7fe449425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.946+0000 7fe449425640 10 monclient(hunting): picked mon.noname-c con 0x7fe4440a9720 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 10 monclient(hunting): picked mon.noname-b con 0x7fe4440ae250 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 10 monclient(hunting): picked mon.noname-a con 0x7fe44414bba0 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44406f740 con 0x7fe44414bba0 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44413f860 con 0x7fe4440ae250 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4441370c0 con 0x7fe4440a9720 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:14.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.948+0000 7fe449425640 10 monclient(hunting): authenticate will time out at 1409.217195s 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.952+0000 7fe442ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe4440a9720 0x7fe444070070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47975/0 (socket says 172.21.15.45:47975) 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.952+0000 7fe4437fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe44414bba0 0x7fe444136990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:48921/0 (socket says 172.21.15.45:48921) 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.952+0000 7fe4437fe640 1 -- 172.21.15.45:0/2774498141 learned_addr learned my addr 172.21.15.45:0/2774498141 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 client.?.objecter ms_handle_connect 0x7fe44414bba0 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 client.?.objecter ms_handle_connect 0x7fe4440a9720 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115860445 0 0) 0x7fe4441370c0 con 0x7fe4440a9720 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.933 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.953+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe430004eb0 con 0x7fe4440a9720 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.954+0000 7fe42b7fe640 10 client.?.objecter ms_handle_connect 0x7fe4440ae250 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.954+0000 7fe42b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888769085 0 0) 0x7fe430004eb0 con 0x7fe4440a9720 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe430005300 con 0x7fe4440a9720 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044306695 0 0) 0x7fe44413f860 con 0x7fe4440ae250 2024-03-29T11:02:14.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe430005760 con 0x7fe4440ae250 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71363301 0 0) 0x7fe44406f740 con 0x7fe44414bba0 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.937 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:02:14.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe44413f860 con 0x7fe44414bba0 2024-03-29T11:02:14.938 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 286224921 0 0) 0x7fe430005760 con 0x7fe4440ae250 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe44406f740 con 0x7fe4440ae250 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe4340016b0 con 0x7fe4440ae250 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): dump: 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.939 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe43c001e80 con 0x7fe4440a9720 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 10 monclient(hunting): dump: 2024-03-29T11:02:14.940 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.955+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 468688092 0 0) 0x7fe430005300 con 0x7fe4440a9720 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 >> v1:172.21.15.116:6789/0 conn(0x7fe4440ae250 legacy=0x7fe444136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.941 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 >> v1:172.21.15.45:6789/0 conn(0x7fe44414bba0 legacy=0x7fe444136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 1 monclient: found mon.c 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44413fcd0 con 0x7fe4440a9720 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 10 monclient: _renew_subs 2024-03-29T11:02:14.942 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44406fc50 con 0x7fe4440a9720 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 10 monclient: _renew_subs 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe444137300 con 0x7fe4440a9720 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe449425640 1 librados: init done 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.956+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe43c003c20 con 0x7fe4440a9720 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.957+0000 7fe42b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.957+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe43c004030 con 0x7fe4440a9720 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.957+0000 7fe42b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.943 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.957+0000 7fe42b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.957+0000 7fe42b7fe640 10 monclient: dump: 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:14.944 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:14.945 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:14.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.957+0000 7fe440ff9640 4 set_mon_vals no callback set 2024-03-29T11:02:14.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.958+0000 7fe449425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.958+0000 7fe449425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.958+0000 7fe449425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:14.945 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.958+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe424005080 con 0x7fe4440a9720 2024-03-29T11:02:14.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.959+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe43c004190 con 0x7fe4440a9720 2024-03-29T11:02:14.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.959+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7fe43c054c20 con 0x7fe4440a9720 2024-03-29T11:02:14.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.959+0000 7fe42b7fe640 10 client.4628.objecter ms_dispatch 0x7fe444000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:14.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.959+0000 7fe42b7fe640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:14.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.959+0000 7fe42b7fe640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:14.946 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.960+0000 7fe42b7fe640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:02:14.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.960+0000 7fe4427fc640 10 monclient: get_auth_request con 0x7fe43006e370 auth_method 0 2024-03-29T11:02:14.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.961+0000 7fe42b7fe640 10 client.4628.objecter ms_handle_connect 0x7fe43006e370 2024-03-29T11:02:14.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.968+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe43c050ec0 con 0x7fe4440a9720 2024-03-29T11:02:14.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.968+0000 7fe42b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:14.968+0000 7fe42b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:15.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.297+0000 7fe449425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:15.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.297+0000 7fe449425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:15.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.297+0000 7fe449425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:15.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.297+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe424005330 con 0x7fe4440a9720 2024-03-29T11:02:15.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.301+0000 7fe42b7fe640 1 -- 172.21.15.45:0/2774498141 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7fe43c051060 con 0x7fe4440a9720 2024-03-29T11:02:15.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.301+0000 7fe42b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:15.281 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.301+0000 7fe42b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:15.283 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:15.283 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:15.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.309+0000 7fe449425640 10 librados: watch_flush enter 2024-03-29T11:02:15.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.309+0000 7fe449425640 10 librados: watch_flush exit 2024-03-29T11:02:15.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.309+0000 7fe449425640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:02:15.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.309+0000 7fe449425640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:02:15.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.310+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe43006e370 legacy=0x7fe430070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:15.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.310+0000 7fe449425640 10 monclient: shutdown 2024-03-29T11:02:15.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.310+0000 7fe449425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:15.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.310+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 >> v1:172.21.15.45:6790/0 conn(0x7fe4440a9720 legacy=0x7fe444070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:15.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.312+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 shutdown_connections 2024-03-29T11:02:15.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.312+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 >> 172.21.15.45:0/2774498141 conn(0x7fe4440b35e0 msgr2=0x7fe444069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:15.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.313+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 shutdown_connections 2024-03-29T11:02:15.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.313+0000 7fe449425640 1 -- 172.21.15.45:0/2774498141 wait complete. 2024-03-29T11:02:15.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:15.313+0000 7fe449425640 1 librados: shutdown 2024-03-29T11:02:15.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:22.118 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:22.261 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.281+0000 7fe10d338640 10 monclient: get_monmap_and_config 2024-03-29T11:02:22.261 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.281+0000 7fe10d338640 10 monclient: build_initial_monmap 2024-03-29T11:02:22.261 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.281+0000 7fe10d338640 10 monclient: monmap: 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:22.282723+0000 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:22.282723+0000 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:22.262 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.283+0000 7fe10d338640 1 Processor -- start 2024-03-29T11:02:22.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.283+0000 7fe10d338640 1 -- start start 2024-03-29T11:02:22.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.283+0000 7fe10d338640 10 monclient: init 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient(hunting): picked mon.noname-b con 0x7fe1080ae490 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient(hunting): picked mon.noname-c con 0x7fe1080aef70 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient(hunting): picked mon.noname-a con 0x7fe1080727a0 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1080b0020 con 0x7fe1080727a0 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10806d960 con 0x7fe1080aef70 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1080ad310 con 0x7fe1080ae490 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:22.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.284+0000 7fe10d338640 10 monclient(hunting): authenticate will time out at 1416.553439s 2024-03-29T11:02:22.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.286+0000 7fe107fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe1080ae490 0x7fe1080713a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39684/0 (socket says 172.21.15.45:39684) 2024-03-29T11:02:22.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.286+0000 7fe107fff640 1 -- 172.21.15.45:0/680404623 learned_addr learned my addr 172.21.15.45:0/680404623 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:22.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.287+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606083966 0 0) 0x7fe1080ad310 con 0x7fe1080ae490 2024-03-29T11:02:22.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.287+0000 7fe106ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:22.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.287+0000 7fe106ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:22.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.287+0000 7fe106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:22.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.287+0000 7fe106ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:02:22.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.287+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f4003520 con 0x7fe1080ae490 2024-03-29T11:02:22.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392841314 0 0) 0x7fe1080b0020 con 0x7fe1080727a0 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1080ad310 con 0x7fe1080727a0 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209816601 0 0) 0x7fe1080ad310 con 0x7fe1080727a0 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 >> v1:172.21.15.45:6790/0 conn(0x7fe1080ae490 legacy=0x7fe1080713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.289+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 >> v1:172.21.15.116:6789/0 conn(0x7fe1080aef70 legacy=0x7fe108072090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 1 monclient: found mon.noname-a 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:02:22.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1081432d0 con 0x7fe1080727a0 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe100001630 con 0x7fe1080727a0 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: dump: 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.271 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe10d338640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe10d338640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe100001880 con 0x7fe1080727a0 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 1 -- 172.21.15.45:0/680404623 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe100001c10 con 0x7fe1080727a0 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe106ffd640 10 monclient: dump: 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:22.272 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe10d338640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:22.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.290+0000 7fe10d338640 4 set_mon_vals no callback set 2024-03-29T11:02:22.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.291+0000 7fe10d338640 10 monclient: shutdown 2024-03-29T11:02:22.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.291+0000 7fe10d338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:22.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.291+0000 7fe10d338640 1 -- 172.21.15.45:0/680404623 >> v1:172.21.15.45:6789/0 conn(0x7fe1080727a0 legacy=0x7fe10814b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.292+0000 7fe10d338640 1 -- 172.21.15.45:0/680404623 shutdown_connections 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.292+0000 7fe10d338640 1 -- 172.21.15.45:0/680404623 >> 172.21.15.45:0/680404623 conn(0x7fe1080673e0 msgr2=0x7fe1080b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.292+0000 7fe10d338640 1 -- 172.21.15.45:0/680404623 shutdown_connections 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.293+0000 7fe10d338640 1 -- 172.21.15.45:0/680404623 wait complete. 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.293+0000 7fe10d338640 10 monclient: build_initial_monmap 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.293+0000 7fe10d338640 10 monclient: monmap: 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:22.276 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 1 librados: starting msgr at 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 1 librados: starting objecter 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 1 Processor -- start 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 1 -- start start 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 1 librados: setting wanted keys 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 1 librados: calling monclient init 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.294+0000 7fe10d338640 10 monclient: init 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient(hunting): picked mon.noname-b con 0x7fe1080727a0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient(hunting): picked mon.noname-a con 0x7fe1080ae490 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient(hunting): picked mon.noname-c con 0x7fe1080aef70 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10813d230 con 0x7fe1080ae490 2024-03-29T11:02:22.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe108145190 con 0x7fe1080727a0 2024-03-29T11:02:22.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10813b300 con 0x7fe1080aef70 2024-03-29T11:02:22.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:22.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.295+0000 7fe10d338640 10 monclient(hunting): authenticate will time out at 1416.564800s 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe1077fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe1080ae490 0x7fe10813dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:53119/0 (socket says 172.21.15.45:53119) 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe1077fe640 1 -- 172.21.15.45:0/3975578901 learned_addr learned my addr 172.21.15.45:0/3975578901 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 client.?.objecter ms_handle_connect 0x7fe1080ae490 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 client.?.objecter ms_handle_connect 0x7fe1080aef70 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879226513 0 0) 0x7fe10813d230 con 0x7fe1080ae490 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:22.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.299+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e8004eb0 con 0x7fe1080ae490 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512852775 0 0) 0x7fe10813b300 con 0x7fe1080aef70 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10813d230 con 0x7fe1080aef70 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 10 client.?.objecter ms_handle_connect 0x7fe1080727a0 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.300+0000 7fe104ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542681212 0 0) 0x7fe10813d230 con 0x7fe1080aef70 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe10813b300 con 0x7fe1080aef70 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81263483 0 0) 0x7fe108145190 con 0x7fe1080727a0 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:22.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10813d230 con 0x7fe1080727a0 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497256641 0 0) 0x7fe0e8004eb0 con 0x7fe1080ae490 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe108145190 con 0x7fe1080ae490 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe100001c10 con 0x7fe1080aef70 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.301+0000 7fe104ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:22.286 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe0f8001ad0 con 0x7fe1080ae490 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.287 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3127648297 0 0) 0x7fe10813b300 con 0x7fe1080aef70 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 >> v1:172.21.15.116:6789/0 conn(0x7fe1080727a0 legacy=0x7fe10806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 >> v1:172.21.15.45:6789/0 conn(0x7fe1080ae490 legacy=0x7fe10813dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 1 monclient: found mon.c 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.302+0000 7fe104ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10814c1e0 con 0x7fe1080aef70 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 10 monclient: _finish_auth 0 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1000038a0 con 0x7fe1080aef70 2024-03-29T11:02:22.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 10 monclient: _renew_subs 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10814b740 con 0x7fe1080aef70 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 10 monclient: _renew_subs 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1081e2f50 con 0x7fe1080aef70 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe10d338640 1 librados: init done 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe100003bb0 con 0x7fe1080aef70 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.303+0000 7fe104ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.304+0000 7fe104ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.304+0000 7fe104ff9640 10 monclient: dump: 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:22.289 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:22.290 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.304+0000 7fe1067fc640 4 set_mon_vals no callback set 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.304+0000 7fe10d338640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.305+0000 7fe10d338640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.305+0000 7fe10d338640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.305+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec005080 con 0x7fe1080aef70 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.305+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe100019b80 con 0x7fe1080aef70 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.308+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7fe1000837a0 con 0x7fe1080aef70 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.308+0000 7fe104ff9640 10 client.4634.objecter ms_dispatch 0x7fe108000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.308+0000 7fe104ff9640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:22.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.308+0000 7fe104ff9640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:22.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.308+0000 7fe104ff9640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:02:22.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.309+0000 7fe107fff640 10 monclient: get_auth_request con 0x7fe0e806de80 auth_method 0 2024-03-29T11:02:22.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.309+0000 7fe104ff9640 10 client.4634.objecter ms_handle_connect 0x7fe0e806de80 2024-03-29T11:02:22.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.314+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe10004f080 con 0x7fe1080aef70 2024-03-29T11:02:22.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.314+0000 7fe104ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:22.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.314+0000 7fe104ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.661+0000 7fe10d338640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.661+0000 7fe10d338640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.661+0000 7fe10d338640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.661+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0ec005330 con 0x7fe1080aef70 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.662+0000 7fe104ff9640 1 -- 172.21.15.45:0/3975578901 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7fe100050fb0 con 0x7fe1080aef70 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.662+0000 7fe104ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:22.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.662+0000 7fe104ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:22.642 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:22.642 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 10 librados: watch_flush enter 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 10 librados: watch_flush exit 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe0e806de80 legacy=0x7fe0e8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 10 monclient: shutdown 2024-03-29T11:02:22.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:22.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 >> v1:172.21.15.45:6790/0 conn(0x7fe1080aef70 legacy=0x7fe10813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 shutdown_connections 2024-03-29T11:02:22.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.666+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 >> 172.21.15.45:0/3975578901 conn(0x7fe1080673e0 msgr2=0x7fe1080b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:22.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.667+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 shutdown_connections 2024-03-29T11:02:22.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.667+0000 7fe10d338640 1 -- 172.21.15.45:0/3975578901 wait complete. 2024-03-29T11:02:22.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:22.667+0000 7fe10d338640 1 librados: shutdown 2024-03-29T11:02:22.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:29.470 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:29.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.649+0000 7fc6ab5c1640 10 monclient: get_monmap_and_config 2024-03-29T11:02:29.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.649+0000 7fc6ab5c1640 10 monclient: build_initial_monmap 2024-03-29T11:02:29.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.649+0000 7fc6ab5c1640 10 monclient: monmap: 2024-03-29T11:02:29.629 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:29.629 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:29.629 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:29.650667+0000 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:29.650667+0000 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.651+0000 7fc6ab5c1640 1 Processor -- start 2024-03-29T11:02:29.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.651+0000 7fc6ab5c1640 1 -- start start 2024-03-29T11:02:29.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.651+0000 7fc6ab5c1640 10 monclient: init 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a41422b0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc6a4142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc6a40b17d0 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a4142c80 con 0x7fc6a40b17d0 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a40aae40 con 0x7fc6a41422b0 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a413fbd0 con 0x7fc6a4142690 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:29.632 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6ab5c1640 10 monclient(hunting): authenticate will time out at 1423.921604s 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.652+0000 7fc6a935d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc6a41422b0 0x7fc6a413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:59926/0 (socket says 172.21.15.45:59926) 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc6a935d640 1 -- 172.21.15.45:0/1160046825 learned_addr learned my addr 172.21.15.45:0/1160046825 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545378099 0 0) 0x7fc6a40aae40 con 0x7fc6a41422b0 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc69bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc69bfff640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.653+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc690003520 con 0x7fc6a41422b0 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461008666 0 0) 0x7fc6a413fbd0 con 0x7fc6a4142690 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a40aae40 con 0x7fc6a4142690 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292075921 0 0) 0x7fc690003520 con 0x7fc6a41422b0 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 >> v1:172.21.15.45:6790/0 conn(0x7fc6a4142690 legacy=0x7fc6a40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 >> v1:172.21.15.45:6789/0 conn(0x7fc6a40b17d0 legacy=0x7fc6a40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 monclient: found mon.noname-c 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a4151940 con 0x7fc6a41422b0 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: _finish_auth 0 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc694001610 con 0x7fc6a41422b0 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: dump: 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc6ab5c1640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:02:29.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc6ab5c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:29.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc694001770 con 0x7fc6a41422b0 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 1 -- 172.21.15.45:0/1160046825 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc694001bc0 con 0x7fc6a41422b0 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.654+0000 7fc69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.655+0000 7fc69bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.655+0000 7fc69bfff640 10 monclient: dump: 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.655+0000 7fc6ab5c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.655+0000 7fc6ab5c1640 4 set_mon_vals no callback set 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 10 monclient: shutdown 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/1160046825 >> v1:172.21.15.116:6789/0 conn(0x7fc6a41422b0 legacy=0x7fc6a413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/1160046825 shutdown_connections 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/1160046825 >> 172.21.15.45:0/1160046825 conn(0x7fc6a4066a00 msgr2=0x7fc6a4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/1160046825 shutdown_connections 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.656+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/1160046825 wait complete. 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.657+0000 7fc6ab5c1640 10 monclient: build_initial_monmap 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.657+0000 7fc6ab5c1640 10 monclient: monmap: 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 1 librados: starting msgr at 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 1 librados: starting objecter 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 1 Processor -- start 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 1 -- start start 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 1 librados: setting wanted keys 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 1 librados: calling monclient init 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.658+0000 7fc6ab5c1640 10 monclient: init 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc6a40b17d0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a41422b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc6a4142690 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a406f4d0 con 0x7fc6a4142690 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a40703a0 con 0x7fc6a40b17d0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a41dbc50 con 0x7fc6a41422b0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6ab5c1640 10 monclient(hunting): authenticate will time out at 1423.928483s 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6a8b5c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc6a41422b0 0x7fc6a41dc170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:44228/0 (socket says 172.21.15.45:44228) 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.659+0000 7fc6a8b5c640 1 -- 172.21.15.45:0/3631912147 learned_addr learned my addr 172.21.15.45:0/3631912147 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 client.?.objecter ms_handle_connect 0x7fc6a41422b0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 client.?.objecter ms_handle_connect 0x7fc6a40b17d0 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 client.?.objecter ms_handle_connect 0x7fc6a4142690 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127601808 0 0) 0x7fc6a41dbc50 con 0x7fc6a41422b0 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc688004eb0 con 0x7fc6a41422b0 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218713245 0 0) 0x7fc6a40703a0 con 0x7fc6a40b17d0 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.660+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a41dbc50 con 0x7fc6a40b17d0 2024-03-29T11:02:29.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289752276 0 0) 0x7fc6a41dbc50 con 0x7fc6a40b17d0 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6a40703a0 con 0x7fc6a40b17d0 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc694001b40 con 0x7fc6a40b17d0 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): dump: 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:29.648 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4142605436 0 0) 0x7fc688004eb0 con 0x7fc6a41422b0 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6a41dbc50 con 0x7fc6a41422b0 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc69c001bf0 con 0x7fc6a41422b0 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.661+0000 7fc699ffb640 10 monclient(hunting): dump: 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.649 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3013359188 0 0) 0x7fc6a40703a0 con 0x7fc6a40b17d0 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 >> v1:172.21.15.45:6790/0 conn(0x7fc6a41422b0 legacy=0x7fc6a41dc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 >> v1:172.21.15.45:6789/0 conn(0x7fc6a4142690 legacy=0x7fc6a413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 1 monclient: found mon.b 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a41dc880 con 0x7fc6a40b17d0 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:29.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 5 monclient: authenticate success, global_id 4594 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 10 monclient: _renew_subs 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a41dcaf0 con 0x7fc6a40b17d0 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 10 monclient: _renew_subs 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a4070c80 con 0x7fc6a40b17d0 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.662+0000 7fc6ab5c1640 1 librados: init done 2024-03-29T11:02:29.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.663+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc694001210 con 0x7fc6a40b17d0 2024-03-29T11:02:29.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.663+0000 7fc699ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:29.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.663+0000 7fc69affd640 4 set_mon_vals no callback set 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fc694004c30 con 0x7fc6a40b17d0 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc699ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc699ffb640 10 monclient: dump: 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.654 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc6ab5c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc6ab5c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc6ab5c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.664+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc684005080 con 0x7fc6a40b17d0 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.666+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fc69401ac00 con 0x7fc6a40b17d0 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc6a8b5c640 10 monclient: get_auth_request con 0x7fc68806dd50 auth_method 0 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7fc694084a80 con 0x7fc6a40b17d0 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc699ffb640 10 client.4594.objecter ms_dispatch 0x7fc6a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:29.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc699ffb640 3 client.4594.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:29.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc699ffb640 3 client.4594.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:29.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc699ffb640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-29T11:02:29.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.667+0000 7fc699ffb640 10 client.4594.objecter ms_handle_connect 0x7fc68806dd50 2024-03-29T11:02:29.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.671+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc694050360 con 0x7fc6a40b17d0 2024-03-29T11:02:29.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.671+0000 7fc699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:29.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:29.671+0000 7fc699ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:30.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.064+0000 7fc6ab5c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:30.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.064+0000 7fc6ab5c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:30.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.064+0000 7fc6ab5c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.116:6789/0 2024-03-29T11:02:30.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.064+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 --> v1:172.21.15.116:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc684005330 con 0x7fc6a40b17d0 2024-03-29T11:02:30.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.066+0000 7fc699ffb640 1 -- 172.21.15.45:0/3631912147 <== mon.1 v1:172.21.15.116:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7fc694052290 con 0x7fc6a40b17d0 2024-03-29T11:02:30.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.066+0000 7fc699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:30.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.066+0000 7fc699ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:30.046 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:30.046 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 10 librados: watch_flush enter 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 10 librados: watch_flush exit 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 10 client.4594.objecter successfully canceled tick 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 >> v1:172.21.15.116:6816/3290579615 conn(0x7fc68806dd50 legacy=0x7fc688070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 10 monclient: shutdown 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 >> v1:172.21.15.116:6789/0 conn(0x7fc6a40b17d0 legacy=0x7fc6a406edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 shutdown_connections 2024-03-29T11:02:30.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 >> 172.21.15.45:0/3631912147 conn(0x7fc6a4066a00 msgr2=0x7fc6a40b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:30.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 shutdown_connections 2024-03-29T11:02:30.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.070+0000 7fc6ab5c1640 1 -- 172.21.15.45:0/3631912147 wait complete. 2024-03-29T11:02:30.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:30.071+0000 7fc6ab5c1640 1 librados: shutdown 2024-03-29T11:02:30.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:36.873 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.046+0000 7efea0d9d640 10 monclient: get_monmap_and_config 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.046+0000 7efea0d9d640 10 monclient: build_initial_monmap 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.046+0000 7efea0d9d640 10 monclient: monmap: 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:37.047362+0000 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:37.047362+0000 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:37.028 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.029 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:37.029 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:37.029 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:37.029 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.049+0000 7efea0d9d640 1 Processor -- start 2024-03-29T11:02:37.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.053+0000 7efea0d9d640 1 -- start start 2024-03-29T11:02:37.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.053+0000 7efea0d9d640 10 monclient: init 2024-03-29T11:02:37.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.054+0000 7efea0d9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:37.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.054+0000 7efea0d9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:37.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.054+0000 7efea0d9d640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0b4100 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:37.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.059+0000 7efea0d9d640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c13ee90 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:37.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.059+0000 7efea0d9d640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c0af900 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:37.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.059+0000 7efea0d9d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0afd10 con 0x7efe9c0af900 2024-03-29T11:02:37.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.059+0000 7efea0d9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0af6f0 con 0x7efe9c0b4100 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.060+0000 7efea0d9d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0aaf60 con 0x7efe9c13ee90 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.060+0000 7efea0d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.060+0000 7efe99d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7efe9c13ee90 0x7efe9c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54991/0 (socket says 172.21.15.45:54991) 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.060+0000 7efe99d9b640 1 -- 172.21.15.45:0/2266554568 learned_addr learned my addr 172.21.15.45:0/2266554568 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.060+0000 7efea0d9d640 10 monclient(hunting): authenticate will time out at 1431.329802s 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.061+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449516590 0 0) 0x7efe9c0aaf60 con 0x7efe9c13ee90 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.061+0000 7efe9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.061+0000 7efe9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.061+0000 7efe9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.061+0000 7efe9959a640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:02:37.041 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.061+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe84003520 con 0x7efe9c13ee90 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064545749 0 0) 0x7efe9c0af6f0 con 0x7efe9c0b4100 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c0aaf60 con 0x7efe9c0b4100 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825202130 0 0) 0x7efe9c0aaf60 con 0x7efe9c0b4100 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 >> v1:172.21.15.45:6790/0 conn(0x7efe9c13ee90 legacy=0x7efe9c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 >> v1:172.21.15.45:6789/0 conn(0x7efe9c0af900 legacy=0x7efe9c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:37.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: _finish_hunting 0 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 monclient: found mon.noname-c 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c13e970 con 0x7efe9c0b4100 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: _finish_auth 0 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: _check_auth_tickets 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efe8c001670 con 0x7efe9c0b4100 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.063+0000 7efe9959a640 10 monclient: dump: 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:37.046 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efea0d9d640 5 monclient: authenticate success, global_id 4597 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efea0d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:37.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe8c001850 con 0x7efe9c0b4100 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efe9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efe9959a640 1 -- 172.21.15.45:0/2266554568 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efe8c001670 con 0x7efe9c0b4100 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efe9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efe9959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efe9959a640 10 monclient: dump: 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:37.048 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efea0d9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.064+0000 7efea0d9d640 4 set_mon_vals no callback set 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.065+0000 7efea0d9d640 10 monclient: shutdown 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.065+0000 7efea0d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.065+0000 7efea0d9d640 1 -- 172.21.15.45:0/2266554568 >> v1:172.21.15.116:6789/0 conn(0x7efe9c0b4100 legacy=0x7efe9c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.065+0000 7efea0d9d640 1 -- 172.21.15.45:0/2266554568 shutdown_connections 2024-03-29T11:02:37.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.065+0000 7efea0d9d640 1 -- 172.21.15.45:0/2266554568 >> 172.21.15.45:0/2266554568 conn(0x7efe9c0b35e0 msgr2=0x7efe9c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.065+0000 7efea0d9d640 1 -- 172.21.15.45:0/2266554568 shutdown_connections 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.066+0000 7efea0d9d640 1 -- 172.21.15.45:0/2266554568 wait complete. 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 10 monclient: build_initial_monmap 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 10 monclient: monmap: 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:37.051 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 1 librados: starting msgr at 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 1 librados: starting objecter 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 1 Processor -- start 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 1 -- start start 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 1 librados: setting wanted keys 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 1 librados: calling monclient init 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.067+0000 7efea0d9d640 10 monclient: init 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.068+0000 7efea0d9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.068+0000 7efea0d9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.068+0000 7efea0d9d640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c0af900 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0b4100 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c13ee90 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:37.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c142660 con 0x7efe9c13ee90 2024-03-29T11:02:37.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c141960 con 0x7efe9c0af900 2024-03-29T11:02:37.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c06e7a0 con 0x7efe9c0b4100 2024-03-29T11:02:37.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:37.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.071+0000 7efea0d9d640 10 monclient(hunting): authenticate will time out at 1431.340558s 2024-03-29T11:02:37.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe99d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7efe9c0b4100 0x7efe9c1deb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55063/0 (socket says 172.21.15.45:55063) 2024-03-29T11:02:37.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe99d9b640 1 -- 172.21.15.45:0/675610825 learned_addr learned my addr 172.21.15.45:0/675610825 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:37.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 10 client.?.objecter ms_handle_connect 0x7efe9c0b4100 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183093453 0 0) 0x7efe9c06e7a0 con 0x7efe9c0b4100 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.074+0000 7efe8affd640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe7c004eb0 con 0x7efe9c0b4100 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190699211 0 0) 0x7efe7c004eb0 con 0x7efe9c0b4100 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9c06e7a0 con 0x7efe9c0b4100 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efe90001870 con 0x7efe9c0b4100 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient(hunting): dump: 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:37.056 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:37.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46594732 0 0) 0x7efe9c06e7a0 con 0x7efe9c0b4100 2024-03-29T11:02:37.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.116:6789/0 conn(0x7efe9c0af900 legacy=0x7efe9c13a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6789/0 conn(0x7efe9c13ee90 legacy=0x7efe9c1e6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 monclient: found mon.c 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c06e930 con 0x7efe9c0b4100 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 client.?.objecter ms_handle_connect 0x7efe9c13ee90 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efe8affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efea0d9d640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efea0d9d640 10 monclient: _renew_subs 2024-03-29T11:02:37.059 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efea0d9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.075+0000 7efea0d9d640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c141c20 con 0x7efe9c0b4100 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe900019d0 con 0x7efe9c0b4100 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efe90004a80 con 0x7efe9c0b4100 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe8affd640 10 monclient: dump: 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:37.060 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.076+0000 7efe98d99640 4 set_mon_vals no callback set 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe99d9b640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6790/0 conn(0x7efe9c0b4100 legacy=0x7efe9c1deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe99d9b640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6790/0 conn(0x7efe9c0b4100 legacy=0x7efe9c1deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe99d9b640 1 --1- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6790/0 conn(0x7efe9c0b4100 0x7efe9c1deb60 :-1 s=OPENED pgs=978325 cs=1 l=1).handle_message read tag failed 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe99d9b640 1 --1- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6790/0 conn(0x7efe9c0b4100 0x7efe9c1deb60 :-1 s=OPENED pgs=978325 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient: ms_handle_reset current mon v1:172.21.15.45:6790/0 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6790/0 conn(0x7efe9c0b4100 legacy=0x7efe9c1deb60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient(hunting): picked mon.a con 0x7efe9c13ee90 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient(hunting): picked mon.c con 0x7efe7c006c80 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient(hunting): picked mon.b con 0x7efe9c0af900 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efe9c06e7a0 con 0x7efe9c13ee90 2024-03-29T11:02:37.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efe9c141960 con 0x7efe9c0af900 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efe7c004eb0 con 0x7efe7c006c80 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efe8affd640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efea0d9d640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efea0d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.077+0000 7efea0d9d640 1 librados: init done 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.079+0000 7efe88ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.079+0000 7efe88ff9640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:37.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.080+0000 7efe8affd640 10 client.4646.objecter ms_handle_connect 0x7efe7c006c80 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.080+0000 7efe8affd640 10 client.4646.objecter resend_mon_ops 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121205176 0 0) 0x7efe7c004eb0 con 0x7efe7c006c80 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7efe7c0139d0 con 0x7efe7c006c80 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 962193100 0 0) 0x7efe7c0139d0 con 0x7efe7c006c80 2024-03-29T11:02:37.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.081+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.116:6789/0 conn(0x7efe9c0af900 legacy=0x7efe7c00fe10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6789/0 conn(0x7efe9c13ee90 legacy=0x7efe7c006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 monclient: found mon.c 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7efe7c014b70 con 0x7efe7c006c80 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c1d28d0 con 0x7efe7c006c80 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c06f8e0 con 0x7efe7c006c80 2024-03-29T11:02:37.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:37.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c06e7a0 con 0x7efe7c006c80 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7efe90005220 con 0x7efe7c006c80 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.084+0000 7efe8affd640 10 monclient: dump: 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:37.069 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.085+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe90005400 con 0x7efe7c006c80 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.085+0000 7efe8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.085+0000 7efe8bfff640 4 set_mon_vals no callback set 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.087+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7efe9001b460 con 0x7efe7c006c80 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.088+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 6 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7efe90085230 con 0x7efe7c006c80 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.088+0000 7efe8affd640 10 client.4646.objecter ms_dispatch 0x7efe9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.088+0000 7efe8affd640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.088+0000 7efe8affd640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.088+0000 7efe8affd640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:02:37.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.089+0000 7efe9a59c640 10 monclient: get_auth_request con 0x7efe9c0b4100 auth_method 0 2024-03-29T11:02:37.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.090+0000 7efe8affd640 10 client.4646.objecter ms_handle_connect 0x7efe9c0b4100 2024-03-29T11:02:37.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.098+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe90050940 con 0x7efe7c006c80 2024-03-29T11:02:37.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.098+0000 7efe8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:37.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.098+0000 7efe8affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:37.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.103+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe90052a40 con 0x7efe7c006c80 2024-03-29T11:02:37.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.103+0000 7efe8affd640 10 monclient: handle_mon_command_ack 1 not found 2024-03-29T11:02:37.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.442+0000 7efe88ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:37.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.442+0000 7efe88ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:37.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.442+0000 7efe88ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:37.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.442+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe9c06e7a0 con 0x7efe7c006c80 2024-03-29T11:02:37.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.443+0000 7efe8affd640 1 -- 172.21.15.45:0/675610825 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7efe90054260 con 0x7efe7c006c80 2024-03-29T11:02:37.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.443+0000 7efe8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:37.423 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.443+0000 7efe8affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:37.423 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:37.423 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 10 librados: watch_flush enter 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 10 librados: watch_flush exit 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.116:6816/3290579615 conn(0x7efe9c0b4100 legacy=0x7efe7c07b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 10 monclient: shutdown 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 >> v1:172.21.15.45:6790/0 conn(0x7efe7c006c80 legacy=0x7efe7c00c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 shutdown_connections 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 >> 172.21.15.45:0/675610825 conn(0x7efe9c0b35e0 msgr2=0x7efe9c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:37.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 shutdown_connections 2024-03-29T11:02:37.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.447+0000 7efe88ff9640 1 -- 172.21.15.45:0/675610825 wait complete. 2024-03-29T11:02:37.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:37.448+0000 7efe88ff9640 1 librados: shutdown 2024-03-29T11:02:37.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:44.250 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.425+0000 7f79b091c640 10 monclient: get_monmap_and_config 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.425+0000 7f79b091c640 10 monclient: build_initial_monmap 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.425+0000 7f79b091c640 10 monclient: monmap: 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:44.426292+0000 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:44.426292+0000 2024-03-29T11:02:44.405 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:44.406 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:44.406 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:44.406 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:44.406 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:44.406 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.426+0000 7f79b091c640 1 Processor -- start 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.426+0000 7f79b091c640 1 -- start start 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.426+0000 7f79b091c640 10 monclient: init 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient(hunting): picked mon.noname-c con 0x7f79ac1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient(hunting): picked mon.noname-b con 0x7f79ac13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient(hunting): picked mon.noname-a con 0x7f79ac0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac0aaef0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac145bd0 con 0x7f79ac1457f0 2024-03-29T11:02:44.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac145eb0 con 0x7f79ac13f650 2024-03-29T11:02:44.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:44.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.427+0000 7f79b091c640 10 monclient(hunting): authenticate will time out at 1438.696601s 2024-03-29T11:02:44.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.429+0000 7f79aa59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f79ac1457f0 0x7f79ac13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:60798/0 (socket says 172.21.15.45:60798) 2024-03-29T11:02:44.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.429+0000 7f79aa59c640 1 -- 172.21.15.45:0/2583883993 learned_addr learned my addr 172.21.15.45:0/2583883993 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:44.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386860083 0 0) 0x7f79ac0aaef0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7994003520 con 0x7f79ac0a9d70 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222155793 0 0) 0x7f79ac145eb0 con 0x7f79ac13f650 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:02:44.412 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ac0aaef0 con 0x7f79ac13f650 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734082903 0 0) 0x7f7994003520 con 0x7f79ac0a9d70 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 >> v1:172.21.15.45:6790/0 conn(0x7f79ac13f650 legacy=0x7f79ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 >> v1:172.21.15.116:6789/0 conn(0x7f79ac1457f0 legacy=0x7f79ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: _finish_hunting 0 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 monclient: found mon.noname-a 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac146340 con 0x7f79ac0a9d70 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: _finish_auth 0 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: _check_auth_tickets 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:44.413 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f79a4001a10 con 0x7f79ac0a9d70 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79a959a640 10 monclient: dump: 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:44.414 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79b091c640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.431+0000 7f79b091c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79a4001c60 con 0x7f79ac0a9d70 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79a959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79a959a640 1 -- 172.21.15.45:0/2583883993 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f79a4001ff0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79a959a640 10 monclient: dump: 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:44.415 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79b091c640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:44.416 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.432+0000 7f79b091c640 4 set_mon_vals no callback set 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.433+0000 7f79b091c640 10 monclient: shutdown 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.433+0000 7f79b091c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.433+0000 7f79b091c640 1 -- 172.21.15.45:0/2583883993 >> v1:172.21.15.45:6789/0 conn(0x7f79ac0a9d70 legacy=0x7f79ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.433+0000 7f79b091c640 1 -- 172.21.15.45:0/2583883993 shutdown_connections 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.433+0000 7f79b091c640 1 -- 172.21.15.45:0/2583883993 >> 172.21.15.45:0/2583883993 conn(0x7f79ac066a00 msgr2=0x7f79ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.435+0000 7f79b091c640 1 -- 172.21.15.45:0/2583883993 shutdown_connections 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.435+0000 7f79b091c640 1 -- 172.21.15.45:0/2583883993 wait complete. 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.436+0000 7f79b091c640 10 monclient: build_initial_monmap 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.436+0000 7f79b091c640 10 monclient: monmap: 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:44.418 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 1 librados: starting msgr at 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 1 librados: starting objecter 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 1 Processor -- start 2024-03-29T11:02:44.419 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 1 -- start start 2024-03-29T11:02:44.420 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 1 librados: setting wanted keys 2024-03-29T11:02:44.420 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 1 librados: calling monclient init 2024-03-29T11:02:44.420 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 10 monclient: init 2024-03-29T11:02:44.420 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:44.420 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.437+0000 7f79b091c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:44.421 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 10 monclient(hunting): picked mon.noname-a con 0x7f79ac0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:44.421 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 10 monclient(hunting): picked mon.noname-c con 0x7f79ac13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:44.421 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 10 monclient(hunting): picked mon.noname-b con 0x7f79ac1457f0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:44.421 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac0add50 con 0x7f79ac0a9d70 2024-03-29T11:02:44.421 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac0acf50 con 0x7f79ac1457f0 2024-03-29T11:02:44.421 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac0ae340 con 0x7f79ac13f650 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79b091c640 10 monclient(hunting): authenticate will time out at 1438.709665s 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79aa59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f79ac0a9d70 0x7f79ac0b0d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:58457/0 (socket says 172.21.15.45:58457) 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.440+0000 7f79aa59c640 1 -- 172.21.15.45:0/2460657816 learned_addr learned my addr 172.21.15.45:0/2460657816 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.441+0000 7f799affd640 10 client.?.objecter ms_handle_connect 0x7f79ac0a9d70 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.441+0000 7f799affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.441+0000 7f799affd640 10 client.?.objecter ms_handle_connect 0x7f79ac1457f0 2024-03-29T11:02:44.422 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.441+0000 7f799affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:44.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339455587 0 0) 0x7f79ac0acf50 con 0x7f79ac1457f0 2024-03-29T11:02:44.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.425 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f798c004eb0 con 0x7f79ac1457f0 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140119693 0 0) 0x7f79ac0add50 con 0x7f79ac0a9d70 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ac0acf50 con 0x7f79ac0a9d70 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 client.?.objecter ms_handle_connect 0x7f79ac13f650 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017089747 0 0) 0x7f79ac0ae340 con 0x7f79ac13f650 2024-03-29T11:02:44.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ac0add50 con 0x7f79ac13f650 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.444+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2329709497 0 0) 0x7f79ac0acf50 con 0x7f79ac0a9d70 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79ac0ae340 con 0x7f79ac0a9d70 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f799c001810 con 0x7f79ac0a9d70 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient(hunting): dump: 2024-03-29T11:02:44.427 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2274407828 0 0) 0x7f79ac0ae340 con 0x7f79ac0a9d70 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 >> v1:172.21.15.45:6790/0 conn(0x7f79ac13f650 legacy=0x7f79ac1da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.428 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 >> v1:172.21.15.116:6789/0 conn(0x7f79ac1457f0 legacy=0x7f79ac1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 monclient: found mon.a 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac0ae4d0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f799affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 10 monclient: _renew_subs 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:44.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac0ad6e0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:44.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 10 monclient: _renew_subs 2024-03-29T11:02:44.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:44.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac1422f0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.445+0000 7f79b091c640 1 librados: init done 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.446+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f799c001d00 con 0x7f79ac0a9d70 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.446+0000 7f799affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.446+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f799c004980 con 0x7f79ac0a9d70 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.446+0000 7f799affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.446+0000 7f799affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.446+0000 7f799affd640 10 monclient: dump: 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:44.431 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.447+0000 7f79a8d99640 4 set_mon_vals no callback set 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.447+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f799c004ae0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.448+0000 7f79b091c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:44.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.448+0000 7f79b091c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.448+0000 7f79b091c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.448+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7f799c0849b0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.448+0000 7f799affd640 10 client.4587.objecter ms_dispatch 0x7f79ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.448+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7990005080 con 0x7f79ac0a9d70 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.449+0000 7f79a9d9b640 10 monclient: get_auth_request con 0x7f798c06dc60 auth_method 0 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.449+0000 7f799affd640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.449+0000 7f799affd640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.449+0000 7f799affd640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:02:44.433 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.449+0000 7f799affd640 10 client.4587.objecter ms_handle_connect 0x7f798c06dc60 2024-03-29T11:02:44.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.456+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f799c0500c0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.456+0000 7f799affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:44.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.456+0000 7f799affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:44.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.784+0000 7f79b091c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:44.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.784+0000 7f79b091c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:44.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.784+0000 7f79b091c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:44.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.784+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7990005330 con 0x7f79ac0a9d70 2024-03-29T11:02:44.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.785+0000 7f799affd640 1 -- 172.21.15.45:0/2460657816 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7f799c0521c0 con 0x7f79ac0a9d70 2024-03-29T11:02:44.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.785+0000 7f799affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:44.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.785+0000 7f799affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:44.765 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:44.765 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:44.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.790+0000 7f79b091c640 10 librados: watch_flush enter 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.790+0000 7f79b091c640 10 librados: watch_flush exit 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.790+0000 7f79b091c640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.790+0000 7f79b091c640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.792+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 >> v1:172.21.15.116:6816/3290579615 conn(0x7f798c06dc60 legacy=0x7f798c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.792+0000 7f79b091c640 10 monclient: shutdown 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.792+0000 7f79b091c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.792+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 >> v1:172.21.15.45:6789/0 conn(0x7f79ac0a9d70 legacy=0x7f79ac0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.793+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 shutdown_connections 2024-03-29T11:02:44.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.793+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 >> 172.21.15.45:0/2460657816 conn(0x7f79ac066a00 msgr2=0x7f79ac13ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:44.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.793+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 shutdown_connections 2024-03-29T11:02:44.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.793+0000 7f79b091c640 1 -- 172.21.15.45:0/2460657816 wait complete. 2024-03-29T11:02:44.773 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:44.794+0000 7f79b091c640 1 librados: shutdown 2024-03-29T11:02:44.792 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:51.600 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:51.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.754+0000 7f379f17c640 10 monclient: get_monmap_and_config 2024-03-29T11:02:51.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.754+0000 7f379f17c640 10 monclient: build_initial_monmap 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.755+0000 7f379f17c640 10 monclient: monmap: 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:51.755860+0000 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:51.755860+0000 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:51.735 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:51.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.756+0000 7f379f17c640 1 Processor -- start 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.756+0000 7f379f17c640 1 -- start start 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.756+0000 7f379f17c640 10 monclient: init 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 10 monclient(hunting): picked mon.noname-c con 0x7f37981457f0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 10 monclient(hunting): picked mon.noname-a con 0x7f379813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379d719640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f37980a9d70 0x7f379814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:42484/0 (socket says 172.21.15.45:42484) 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379cf18640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f37981457f0 0x7f379813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:39218/0 (socket says 172.21.15.45:39218) 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379d719640 1 -- 172.21.15.45:0/1494345961 learned_addr learned my addr 172.21.15.45:0/1494345961 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 10 monclient(hunting): picked mon.noname-b con 0x7f37980a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980aaef0 con 0x7f379813f650 2024-03-29T11:02:51.741 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.757+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798145bd0 con 0x7f37981457f0 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798145eb0 con 0x7f37980a9d70 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f379f17c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f379f17c640 10 monclient(hunting): authenticate will time out at 1446.027313s 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249030924 0 0) 0x7f3798145bd0 con 0x7f37981457f0 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3788003520 con 0x7f37981457f0 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320960840 0 0) 0x7f3798145eb0 con 0x7f37980a9d70 2024-03-29T11:02:51.742 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798145bd0 con 0x7f37980a9d70 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749054139 0 0) 0x7f3798145bd0 con 0x7f37980a9d70 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 >> v1:172.21.15.116:6789/0 conn(0x7f37981457f0 legacy=0x7f379813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.758+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 >> v1:172.21.15.45:6789/0 conn(0x7f379813f650 legacy=0x7f37980b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 1 monclient: found mon.noname-b 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3798146340 con 0x7f37980a9d70 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:51.743 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3790001970 con 0x7f37980a9d70 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: dump: 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f379f17c640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:02:51.744 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f379f17c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3790001b90 con 0x7f37980a9d70 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 1 -- 172.21.15.45:0/1494345961 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3790001f20 con 0x7f37980a9d70 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.759+0000 7f37977fe640 10 monclient: dump: 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.745 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.760+0000 7f379f17c640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.760+0000 7f379f17c640 4 set_mon_vals no callback set 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.761+0000 7f379f17c640 10 monclient: shutdown 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.761+0000 7f379f17c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.761+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 >> v1:172.21.15.45:6790/0 conn(0x7f37980a9d70 legacy=0x7f379814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.761+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 shutdown_connections 2024-03-29T11:02:51.747 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.761+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 >> 172.21.15.45:0/1494345961 conn(0x7f3798066a00 msgr2=0x7f3798066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.762+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 shutdown_connections 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.762+0000 7f379f17c640 1 -- 172.21.15.45:0/1494345961 wait complete. 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.762+0000 7f379f17c640 10 monclient: build_initial_monmap 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.762+0000 7f379f17c640 10 monclient: monmap: 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:51.754 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 1 librados: starting msgr at 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 1 librados: starting objecter 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 1 Processor -- start 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 1 -- start start 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 1 librados: setting wanted keys 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 1 librados: calling monclient init 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.763+0000 7f379f17c640 10 monclient: init 2024-03-29T11:02:51.755 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.764+0000 7f379f17c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.764+0000 7f379f17c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 10 monclient(hunting): picked mon.noname-a con 0x7f37980a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 10 monclient(hunting): picked mon.noname-c con 0x7f379813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 10 monclient(hunting): picked mon.noname-b con 0x7f37981457f0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798142170 con 0x7f37980a9d70 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980abeb0 con 0x7f37981457f0 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980ac120 con 0x7f379813f650 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.766+0000 7f379f17c640 10 monclient(hunting): authenticate will time out at 1446.035398s 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f3797fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f379813f650 0x7f37981370f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49042/0 (socket says 172.21.15.45:49042) 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f3797fff640 1 -- 172.21.15.45:0/2191706816 learned_addr learned my addr 172.21.15.45:0/2191706816 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 10 client.?.objecter ms_handle_connect 0x7f379813f650 2024-03-29T11:02:51.756 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055395325 0 0) 0x7f37980ac120 con 0x7f379813f650 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.767+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c004eb0 con 0x7f379813f650 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55029451 0 0) 0x7f377c004eb0 con 0x7f379813f650 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37980ac120 con 0x7f379813f650 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3784001960 con 0x7f379813f650 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): dump: 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:51.757 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 client.?.objecter ms_handle_connect 0x7f37980a9d70 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 client.?.objecter ms_handle_connect 0x7f37981457f0 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:51.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504761179 0 0) 0x7f37980abeb0 con 0x7f37981457f0 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c004eb0 con 0x7f37981457f0 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832673486 0 0) 0x7f3798142170 con 0x7f37980a9d70 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.768+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37980abeb0 con 0x7f37980a9d70 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658412465 0 0) 0x7f37980abeb0 con 0x7f37980a9d70 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3798142170 con 0x7f37980a9d70 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 790153983 0 0) 0x7f37980ac120 con 0x7f379813f650 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 >> v1:172.21.15.116:6789/0 conn(0x7f37981457f0 legacy=0x7f37981e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 >> v1:172.21.15.45:6789/0 conn(0x7f37980a9d70 legacy=0x7f3798142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 monclient: found mon.c 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37980ac400 con 0x7f379813f650 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: _finish_auth 0 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 10 monclient: _renew_subs 2024-03-29T11:02:51.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3798142d60 con 0x7f379813f650 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 10 monclient: _renew_subs 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f379813ff80 con 0x7f379813f650 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f379f17c640 1 librados: init done 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3784001c80 con 0x7f379813f650 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7f3784003a40 con 0x7f379813f650 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.769+0000 7f37957fa640 10 monclient: dump: 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:51.761 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.771+0000 7f379f17c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.771+0000 7f3796ffd640 4 set_mon_vals no callback set 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.772+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7f3784003ba0 con 0x7f379813f650 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.772+0000 7f379f17c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:51.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.772+0000 7f379f17c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:51.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.772+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3798075f10 con 0x7f379813f650 2024-03-29T11:02:51.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.773+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7f3784084a60 con 0x7f379813f650 2024-03-29T11:02:51.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.773+0000 7f37957fa640 10 client.4658.objecter ms_dispatch 0x7f3798000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:51.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.773+0000 7f37957fa640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:51.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.773+0000 7f37957fa640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:51.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.773+0000 7f37957fa640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T11:02:51.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.775+0000 7f379cf18640 10 monclient: get_auth_request con 0x7f377c076760 auth_method 0 2024-03-29T11:02:51.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.775+0000 7f37957fa640 10 client.4658.objecter ms_handle_connect 0x7f377c076760 2024-03-29T11:02:51.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.779+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3784050340 con 0x7f379813f650 2024-03-29T11:02:51.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.779+0000 7f37957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:51.764 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:51.779+0000 7f37957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:52.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.093+0000 7f379f17c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:52.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.094+0000 7f379f17c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:52.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.094+0000 7f379f17c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:02:52.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.094+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37980b3680 con 0x7f379813f650 2024-03-29T11:02:52.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.094+0000 7f37957fa640 1 -- 172.21.15.45:0/2191706816 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7f3784052270 con 0x7f379813f650 2024-03-29T11:02:52.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.094+0000 7f37957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:52.074 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.094+0000 7f37957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:52.074 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:52.075 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:52.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.097+0000 7f379f17c640 10 librados: watch_flush enter 2024-03-29T11:02:52.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.097+0000 7f379f17c640 10 librados: watch_flush exit 2024-03-29T11:02:52.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.097+0000 7f379f17c640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T11:02:52.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.097+0000 7f379f17c640 10 client.4658.objecter successfully canceled tick 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 >> v1:172.21.15.116:6816/3290579615 conn(0x7f377c076760 legacy=0x7f377c078c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 10 monclient: shutdown 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 >> v1:172.21.15.45:6790/0 conn(0x7f379813f650 legacy=0x7f37981370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 shutdown_connections 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 >> 172.21.15.45:0/2191706816 conn(0x7f3798066a00 msgr2=0x7f3798066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 shutdown_connections 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 -- 172.21.15.45:0/2191706816 wait complete. 2024-03-29T11:02:52.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:52.098+0000 7f379f17c640 1 librados: shutdown 2024-03-29T11:02:52.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:58.899 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:59.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.068+0000 7fe51f274640 10 monclient: get_monmap_and_config 2024-03-29T11:02:59.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.068+0000 7fe51f274640 10 monclient: build_initial_monmap 2024-03-29T11:02:59.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.068+0000 7fe51f274640 10 monclient: monmap: 2024-03-29T11:02:59.048 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:02:59.069870+0000 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:02:59.069870+0000 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-c 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:02:59.049 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.069+0000 7fe51f274640 1 Processor -- start 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.070+0000 7fe51f274640 1 -- start start 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.070+0000 7fe51f274640 10 monclient: init 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.071+0000 7fe51f274640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.071+0000 7fe51f274640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.071+0000 7fe51f274640 10 monclient(hunting): picked mon.noname-b con 0x7fe5180b0860 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.071+0000 7fe51f274640 10 monclient(hunting): picked mon.noname-a con 0x7fe51813ef30 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.072+0000 7fe51f274640 10 monclient(hunting): picked mon.noname-c con 0x7fe5180af8e0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.072+0000 7fe51f274640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5180b0c40 con 0x7fe51813ef30 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.072+0000 7fe51f274640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5180aed10 con 0x7fe5180af8e0 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.072+0000 7fe51f274640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5180afd50 con 0x7fe5180b0860 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.072+0000 7fe51f274640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:59.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.072+0000 7fe51f274640 10 monclient(hunting): authenticate will time out at 1453.342264s 2024-03-29T11:02:59.054 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.073+0000 7fe51d811640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe5180af8e0 0x7fe518149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:56366/0 (socket says 172.21.15.45:56366) 2024-03-29T11:02:59.054 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.073+0000 7fe51d811640 1 -- 172.21.15.45:0/2919878319 learned_addr learned my addr 172.21.15.45:0/2919878319 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:59.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072620927 0 0) 0x7fe5180aed10 con 0x7fe5180af8e0 2024-03-29T11:02:59.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:59.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:59.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:59.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:02:59.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe504003520 con 0x7fe5180af8e0 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2406187837 0 0) 0x7fe504003520 con 0x7fe5180af8e0 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.076+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 >> v1:172.21.15.45:6790/0 conn(0x7fe5180b0860 legacy=0x7fe51813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 >> v1:172.21.15.45:6789/0 conn(0x7fe51813ef30 legacy=0x7fe5180af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 1 monclient: found mon.noname-c 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.116:6789/0 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5180ab1b0 con 0x7fe5180af8e0 2024-03-29T11:02:59.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: _finish_auth 0 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe510001a10 con 0x7fe5180af8e0 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.077+0000 7fe50ffff640 10 monclient: dump: 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 <== mon.1 v1:172.21.15.116:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe510001ca0 con 0x7fe5180af8e0 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe50ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe50ffff640 1 -- 172.21.15.45:0/2919878319 <== mon.1 v1:172.21.15.116:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe510002070 con 0x7fe5180af8e0 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe50ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe50ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe50ffff640 10 monclient: dump: 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe51f274640 5 monclient: authenticate success, global_id 4609 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe51f274640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.078+0000 7fe51f274640 4 set_mon_vals no callback set 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 10 monclient: shutdown 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 1 -- 172.21.15.45:0/2919878319 >> v1:172.21.15.116:6789/0 conn(0x7fe5180af8e0 legacy=0x7fe518149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 1 -- 172.21.15.45:0/2919878319 shutdown_connections 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 1 -- 172.21.15.45:0/2919878319 >> 172.21.15.45:0/2919878319 conn(0x7fe51806b660 msgr2=0x7fe51806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:59.063 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 1 -- 172.21.15.45:0/2919878319 shutdown_connections 2024-03-29T11:02:59.064 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.079+0000 7fe51f274640 1 -- 172.21.15.45:0/2919878319 wait complete. 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.082+0000 7fe51f274640 10 monclient: build_initial_monmap 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.082+0000 7fe51f274640 10 monclient: monmap: 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.noname-b 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.082+0000 7fe51f274640 1 librados: starting msgr at 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.082+0000 7fe51f274640 1 librados: starting objecter 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.084+0000 7fe51f274640 1 Processor -- start 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.084+0000 7fe51f274640 1 -- start start 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.084+0000 7fe51f274640 1 librados: setting wanted keys 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.084+0000 7fe51f274640 1 librados: calling monclient init 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.084+0000 7fe51f274640 10 monclient: init 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.085+0000 7fe51f274640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:59.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.085+0000 7fe51f274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:59.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.085+0000 7fe51f274640 10 monclient(hunting): picked mon.noname-b con 0x7fe5180af8e0 addr v1:172.21.15.116:6789/0 2024-03-29T11:02:59.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe51d010640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe5180af8e0 0x7fe5181d8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.45:58560/0 (socket says 172.21.15.45:58560) 2024-03-29T11:02:59.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe51c80f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe5180b0860 0x7fe518070250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:36297/0 (socket says 172.21.15.45:36297) 2024-03-29T11:02:59.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe51d010640 1 -- 172.21.15.45:0/3622007059 learned_addr learned my addr 172.21.15.45:0/3622007059 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:02:59.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe50dffb640 10 client.?.objecter ms_handle_connect 0x7fe5180af8e0 2024-03-29T11:02:59.069 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe50dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe50dffb640 10 client.?.objecter ms_handle_connect 0x7fe5180b0860 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.088+0000 7fe50dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 10 monclient(hunting): picked mon.noname-a con 0x7fe5180b0860 addr v1:172.21.15.45:6789/0 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 10 monclient(hunting): picked mon.noname-c con 0x7fe51813ef30 addr v1:172.21.15.45:6790/0 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5181d9320 con 0x7fe5180b0860 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe518138600 con 0x7fe5180af8e0 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51806db00 con 0x7fe51813ef30 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:59.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.089+0000 7fe51f274640 10 monclient(hunting): authenticate will time out at 1453.359490s 2024-03-29T11:02:59.072 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133883590 0 0) 0x7fe518138600 con 0x7fe5180af8e0 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc004eb0 con 0x7fe5180af8e0 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919084630 0 0) 0x7fe5181d9320 con 0x7fe5180b0860 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:02:59.073 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.092+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe518138600 con 0x7fe5180b0860 2024-03-29T11:02:59.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 client.?.objecter ms_handle_connect 0x7fe51813ef30 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3065768514 0 0) 0x7fe4fc004eb0 con 0x7fe5180af8e0 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.116:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5181d9320 con 0x7fe5180af8e0 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039710403 0 0) 0x7fe518138600 con 0x7fe5180b0860 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4fc004eb0 con 0x7fe5180b0860 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe514001990 con 0x7fe5180af8e0 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): dump: 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.078 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe508001810 con 0x7fe5180b0860 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): dump: 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.079 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312997731 0 0) 0x7fe51806db00 con 0x7fe51813ef30 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe518138600 con 0x7fe51813ef30 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1616455745 0 0) 0x7fe4fc004eb0 con 0x7fe5180b0860 2024-03-29T11:02:59.080 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 >> v1:172.21.15.45:6790/0 conn(0x7fe51813ef30 legacy=0x7fe5181364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 >> v1:172.21.15.116:6789/0 conn(0x7fe5180af8e0 legacy=0x7fe5181d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 monclient: found mon.a 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5181d95f0 con 0x7fe5180b0860 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe50dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe51f274640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe51f274640 10 monclient: _renew_subs 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe51f274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.093+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe518070f10 con 0x7fe5180b0860 2024-03-29T11:02:59.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe51f274640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe51f274640 10 monclient: _renew_subs 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe51f274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe518070960 con 0x7fe5180b0860 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe508003610 con 0x7fe5180b0860 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe51f274640 1 librados: init done 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe50dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.094+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2637228799 0 0) 0x7fe508001970 con 0x7fe5180b0860 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.095+0000 7fe50f7fe640 4 set_mon_vals no callback set 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 10 monclient: dump: 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:fsid 71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017 2024-03-29T11:02:59.082 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T10:51:25.243898+0000 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.116:6789/0 mon.b 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1689116033 0 0) 0x7fe508003960 con 0x7fe5180b0860 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1631295920 0 0) 0x7fe5080848a0 con 0x7fe5180b0860 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 10 client.4596.objecter ms_dispatch 0x7fe518000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe50dffb640 3 client.4596.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.096+0000 7fe51d010640 10 monclient: get_auth_request con 0x7fe4fc072460 auth_method 0 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe50dffb640 3 client.4596.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe50dffb640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe50dffb640 10 client.4596.objecter ms_handle_connect 0x7fe4fc072460 2024-03-29T11:02:59.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe51f274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:59.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe51f274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:59.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe51f274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:59.084 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.097+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f8005080 con 0x7fe5180b0860 2024-03-29T11:02:59.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.104+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe508050180 con 0x7fe5180b0860 2024-03-29T11:02:59.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.104+0000 7fe50dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:59.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.104+0000 7fe50dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:59.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.423+0000 7fe51f274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:59.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.423+0000 7fe51f274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:59.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.423+0000 7fe51f274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:02:59.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.423+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4f8005330 con 0x7fe5180b0860 2024-03-29T11:02:59.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.424+0000 7fe50dffb640 1 -- 172.21.15.45:0/3622007059 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 624972924) 0x7fe5080520b0 con 0x7fe5180b0860 2024-03-29T11:02:59.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.425+0000 7fe50dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:59.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.425+0000 7fe50dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:59.405 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:02:59.405 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"71ecc2b4-4c13-4ddd-b5df-40aa1dcc9017","created":"2024-03-29T10:52:01.478705+0000","modified":"2024-03-29T11:02:06.393878+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:02:06.393878+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383bc44-6790-4a17-9217-d8bfe1f9974a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"105e5d19-e910-4a79-92e6-56d6a0706900","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5fa13e47-ddf2-43fb-87c2-1e22289c6cf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aacf3eb4-f626-4306-9973-92189256dbec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"da3ec8c9-8243-4f39-b6a1-3e04c9ef6fa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5645e2a-7d55-44f9-946e-18a6d8dd20d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4cf69c62-224b-49d1-8c83-cf9cec6104af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8fa620d6-caba-4277-bb1c-aa078e0fac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:59.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.428+0000 7fe51f274640 10 librados: watch_flush enter 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 10 librados: watch_flush exit 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 >> v1:172.21.15.116:6816/3290579615 conn(0x7fe4fc072460 legacy=0x7fe4fc074920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 10 monclient: shutdown 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 >> v1:172.21.15.45:6789/0 conn(0x7fe5180b0860 legacy=0x7fe518070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 shutdown_connections 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 >> 172.21.15.45:0/3622007059 conn(0x7fe51806b660 msgr2=0x7fe51813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 shutdown_connections 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.430+0000 7fe51f274640 1 -- 172.21.15.45:0/3622007059 wait complete. 2024-03-29T11:02:59.411 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:02:59.431+0000 7fe51f274640 1 librados: shutdown 2024-03-29T11:02:59.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:59.427 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:02:59.429 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:02:59.429 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:02:59.429 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.468 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:02:59.468 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:02:59.469 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.545 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:02:59.545 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:02:59.545 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.630 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:02:59.630 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:02:59.630 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.670 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:02:59.670 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:02:59.670 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.714 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:02:59.714 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:02:59.714 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.749 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:02:59.749 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:02:59.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.816 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:02:59.816 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:02:59.817 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.886 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:02:59.886 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:02:59.887 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:02:59.887 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.910 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:02:59.910 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:02:59.911 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.951 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:02:59.952 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:02:59.952 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:02:59.952 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:59.963 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:02:59.963 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:02:59.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:03:00.014 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:03:00.015 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:03:00.015 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:03:00.025 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:03:00.026 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:03:00.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:03:00.086 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:03:00.086 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:03:03.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:03:03.401 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:03:03.528 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:03:03.528 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:03:03.619 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:03:03.619 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:03:03.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T11:03:03.678 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:03:06.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T11:03:06.750 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:03:06.819 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T11:03:06.820 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:03:06.876 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T11:03:06.877 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:03:06.942 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:03:06.943 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/data/mon.a.tgz 2024-03-29T11:03:06.944 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-29T11:03:06.962 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.JGvbbVMAR2 2024-03-29T11:03:06.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.JGvbbVMAR2 2024-03-29T11:03:07.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.JGvbbVMAR2 2024-03-29T11:03:07.157 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.JGvbbVMAR2 is 127KB 2024-03-29T11:03:07.209 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.JGvbbVMAR2 2024-03-29T11:03:07.224 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/data/mon.c.tgz 2024-03-29T11:03:07.224 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-29T11:03:07.278 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.4BKOUz5mNr 2024-03-29T11:03:07.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4BKOUz5mNr 2024-03-29T11:03:07.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.4BKOUz5mNr 2024-03-29T11:03:07.452 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.4BKOUz5mNr is 127KB 2024-03-29T11:03:07.503 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.4BKOUz5mNr 2024-03-29T11:03:07.518 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/data/mon.b.tgz 2024-03-29T11:03:07.518 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-29T11:03:07.536 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.6OX95mH4wj 2024-03-29T11:03:07.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6OX95mH4wj 2024-03-29T11:03:07.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.6OX95mH4wj 2024-03-29T11:03:07.735 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.6OX95mH4wj is 127KB 2024-03-29T11:03:07.786 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.6OX95mH4wj 2024-03-29T11:03:07.801 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:03:07.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:03:07.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:03:07.884 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:03:07.923 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:03:07.923 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:03:07.925 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/remote/smithi045/crash 2024-03-29T11:03:07.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:03:07.953 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/remote/smithi116/crash 2024-03-29T11:03:07.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:03:07.989 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:03:07.989 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:03:07.997 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:03:08.060 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40104.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41723.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40026.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.a.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40104.log.gz 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41762.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40026.log.gz 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38776.log 2024-03-29T11:03:08.061 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41762.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40377.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ceph-client.admin.38776.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38581.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38776.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40377.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39870.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38581.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40416.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-03-29T11:03:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40416.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39870.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40416.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40494.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38620.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41357.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-03-29T11:03:08.063 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40494.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41005.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41357.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39246.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39675.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39246.log.gz 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40611.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40611.log: gzip 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz -5 2024-03-29T11:03:08.064 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8%/var/log/ceph/ceph-client.admin.39363.log: -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41957.log 85.8% 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40182.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41957.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41879.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41957.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42035.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40182.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41840.log 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-03-29T11:03:08.065 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.42035.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41879.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39753.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42035.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39714.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41840.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39753.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38856.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39753.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39714.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39480.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38856.log.gz 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40260.log 2024-03-29T11:03:08.066 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39987.log: /var/log/ceph/ceph-client.admin.40260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39519.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40260.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40927.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41318.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39402.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39831.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35371.log 2024-03-29T11:03:08.067 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41318.log.gz 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40807.log: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.39831.log.gz -5 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34977.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35371.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.34977.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35371.log.gz 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40650.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41996.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34977.log.gz 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41489.log 2024-03-29T11:03:08.068 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41489.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29146.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40729.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41489.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39207.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.tmp-client.admin.29146.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29146.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39207.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40338.log 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39207.log.gz 2024-03-29T11:03:08.069 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38659.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38934.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38659.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40338.log.gz 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40572.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38934.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38659.log.gz 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38934.log.gz 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40572.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr: 85.4%/var/log/ceph/ceph-client.admin.41200.log: -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 85.9% 2024-03-29T11:03:08.070 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41918.log: 85.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41450.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41450.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40966.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41450.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.41801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39129.log 2024-03-29T11:03:08.071 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35268.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39597.log: 93.8% 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39129.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39168.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40299.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:gzip 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40299.log: /var/log/ceph/ceph-client.admin.39558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41278.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40299.log.gz 2024-03-29T11:03:08.072 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39090.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41239.log: /var/log/ceph/ceph-client.admin.40221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38737.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39090.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40533.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38737.log.gz 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2024-03-29T11:03:08.073 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz 2024-03-29T11:03:08.074 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6%/var/log/ceph/ceph-client.admin.39285.log: -- replaced with /var/log/ceph/ceph-client.admin.40533.log.gz 2024-03-29T11:03:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:03:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40690.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2024-03-29T11:03:08.074 INFO:teuthology.orchestra.run.smithi045.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2024-03-29T11:03:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40885.log 2024-03-29T11:03:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.40885.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40885.log.gz 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34529.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34609.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34529.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34609.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34529.log.gz 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34609.log.gz 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:03:08.080 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34569.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.7.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34649.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34449.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34489.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34649.log.gz 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34449.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34489.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34449.log.gz 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34489.log.gz 2024-03-29T11:03:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:03:08.112 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:03:19.590 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:03:19.760 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:03:19.995 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:03:20.809 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:03:22.890 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:03:23.692 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:03:24.134 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:03:24.492 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:03:57.884 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:03:59.103 INFO:teuthology.orchestra.run.smithi045.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:03:59.105 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:03:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m51.095s 2024-03-29T11:03:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:user 2m32.898s 2024-03-29T11:03:59.105 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m5.054s 2024-03-29T11:04:16.918 INFO:teuthology.orchestra.run.smithi116.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:04:16.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-29T11:04:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:real 1m8.872s 2024-03-29T11:04:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:user 1m58.082s 2024-03-29T11:04:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m4.265s 2024-03-29T11:04:16.923 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:04:16.924 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/remote/smithi045/log 2024-03-29T11:04:16.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:04:38.472 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/remote/smithi116/log 2024-03-29T11:04:38.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:04:56.817 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:04:57.022 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bae6fe412339420bb735c1bd76dbe5e0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:04:57.026 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:04:57.034 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:04:57.052 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:04:57.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:04:57.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:04:57.146 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:04:57.155 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:04:57.155 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:04:57.158 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:04:57.172 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:04:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:04:57.178 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:04:57.178 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-29T11:04:57.178 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +414us[ +414us] +/- 67ms 2024-03-29T11:04:57.178 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -193us[ -193us] +/- 62ms 2024-03-29T11:04:57.178 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -192us[ -192us] +/- 92ms 2024-03-29T11:04:57.178 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:04:57.179 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:04:57.179 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-29T11:04:57.180 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +393us[ +408us] +/- 67ms 2024-03-29T11:04:57.180 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -206us[ -191us] +/- 62ms 2024-03-29T11:04:57.180 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -188us[ -188us] +/- 92ms 2024-03-29T11:04:57.180 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:04:57.180 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:04:57.188 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:04:57.189 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:04:57.196 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:04:57.289 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:04:57.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:04:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:57.390 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:04:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:57.421 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:04:57.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:04:57.438 INFO:teuthology.task.internal:Duration was 1215.439857 seconds 2024-03-29T11:04:57.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:04:57.446 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:04:57.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:04:57.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:04:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:04:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:04:57.937 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:04:57.938 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:04:57.938 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:04:57.958 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-29T11:04:57.958 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:04:57.982 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:04:57.983 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:04:58.001 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:04:58.051 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:04:58.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:04:58.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:04:58.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:04:58.199 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:04:58.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:04:58.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:04:58.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:04:58.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:58.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:58.290 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-03-29T11:04:58.318 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-29T11:04:58.363 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:58.377 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:58.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:04:58.427 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:04:58.429 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/remote/smithi045 2024-03-29T11:04:58.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:04:58.465 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660/remote/smithi116 2024-03-29T11:04:58.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:04:58.501 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:04:58.501 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:04:58.503 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:04:58.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:04:58.562 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:04:58.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:04:58.571 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:04:58.571 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:04:58.574 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:04:58.585 INFO:teuthology.orchestra.run.smithi045.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:04 /home/ubuntu/cephtest 2024-03-29T11:04:58.604 INFO:teuthology.orchestra.run.smithi116.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:04 /home/ubuntu/cephtest 2024-03-29T11:04:58.605 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:04:58.615 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:04:58.637 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:04:58.661 DEBUG:teuthology.nuke:shortname: smithi045 2024-03-29T11:04:58.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:04:58.688 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:36:10.429673', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI4p6ibs7lJ1PiB5ycWMubtAd2eIXTbxBFYTi66d687eW0XvLop/s7OH1CsTcIyd817+s3Z3oSShrHyRMeRBVPs='} 2024-03-29T11:04:58.692 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-29T11:04:58.692 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:04:58.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622660', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:36:10.433228', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBVO7ruRl0xSWBFQuwtmzxUOeWSAOQUBI3q2WwjDZWo+Z/fgFJpv8+BTJQkRGHrN+748R3CHxJKpIbVn/j9Vn7g='} 2024-03-29T11:04:58.741 INFO:teuthology.orchestra.console.smithi045:Power off 2024-03-29T11:04:58.742 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:04:58.764 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-03-29T11:04:58.764 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:58.781 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-29T11:04:58.782 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:04:58.791 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-29T11:04:58.804 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-29T11:04:58.804 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:58.827 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-29T11:05:02.793 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:02.828 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:02.903 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-29T11:05:02.938 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-29T11:05:06.905 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:06.939 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:07.015 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-03-29T11:05:07.049 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-29T11:05:07.117 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-03-29T11:05:11.052 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:11.163 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-29T11:05:11.264 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-29T11:05:11.368 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1215.4398572444916 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bae6fe412339420bb735c1bd76dbe5e0 status: fail success: false 2024-03-29T11:05:11.368 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:05:11.449 INFO:teuthology.run:FAIL